Assigning IP addresses to Docker containers via DHCP

I want to run a DHCP server on each of the hosts. It will be responsible for assigning the IPs to all of the containers connected to the docker0 bridge on that host.
You may ask why I want to run so many DHCP servers? Of course one server would do the job of assigning the IP’s (in fact, this was my initial set-up). Then I realized that it would be easier to manage routing of the container’s traffic if we have a DHCP server on every host.
Since we want to have the containers run on the same network, there will be many DHCP servers working onthat network. The trick is to reserve a bunch of IP’s for each host so as not to interfere with other hosts (in my case it’s 10 addresses per host) and drop the DHCP requests on each br0 Open vSwitch bridge.

New scripts

Additionally I rewrote my earlier script that prepares the virtual machines for me. I decided to drop the use ofcloud-init entirely in favor of libguestfs — a great Swiss Army knife written by Rich. If you’re not familiar with it — it’s a tool for offline manipulation of disk images. And it does the job damn well.
You can install libguestfs tools on your system by running this command:
yum -y install libguestfs-tools
Install liguestfs-tools before you use my scripts.
Note
The first run of guestfish can take a bit, since it creates the supermin appliance. Also, if you run it in a virtualized environment it will perform a bit worse compared to bare metal.
The new scripts are avilable in the docker-dhcp repository on GitHub.
The logic of preparing and registering the image as a libvirt domain was split into two scripts:
The split made it possible to create the image once and create as many domains as you want in seconds.
I placed a lot of comments in these scripts, so I hope everything is self-explaining. Let me know if that’s not the case!

Base image

To create a VM with everything preinstalled to make the setup easier the only thing you need to do is to download the QCOW2 image from cloud.fedoraproject.org and run the script providing the cloud image as the parameter, like this:
$ ./prepare-image.sh Fedora-x86_64-20-20131211.1-sda.qcow2
Wed, 29 Jan 2014 12:20:57 +0100 Cleaniung up...
Wed, 29 Jan 2014 12:20:58 +0100 Modifying the image...
Wed, 29 Jan 2014 12:25:15 +0100 Resizing the disk...
Wed, 29 Jan 2014 12:26:06 +0100 Image 'image.qcow2' created!
Note
Executing the above script can take some time. It took about 5 minutes for me to prepare the image. Please keep in mind that it does the full system update and it installs some additional software as well.
This script injects the network.sh script which will be used later to configure the network inside the hosts.

Host domains

Now we have the base image prepared: image.qcow2. Time to make use of it and register two domains based on it. For this purpose I use the register-domain.sh script:
$ ./register-domain.sh image.qcow2 host1
Wed, 29 Jan 2014 13:52:51 +0100 Installing the domain and adjusting the configuration...
Wed, 29 Jan 2014 13:52:51 +0100 Domain host1 registered!
Wed, 29 Jan 2014 13:52:51 +0100 Launching the host1 domain...
Wed, 29 Jan 2014 13:52:52 +0100 Domain started, waiting for the IP...
Wed, 29 Jan 2014 13:53:30 +0100 You can ssh to the 192.168.122.144 host using 'fedora' username and 'fedora' password or use the 'virsh console host1' command to directly attach to the console
You can log-in using the fedora / fedora credentials.
Note
If you don’t want to run the domain immediately after creation — use the RUN_AFTER environment variable and set it to false.
Run the register-domain.sh script twice with host1 and host2 as the arguments.

DHCP configuration explained

The DHCP server will be run on every host and listen only for requests on the docker0 interface since it’s configured to look for the 172.16.42.0/24 network only. The first 9 IP addresses from this network are reserved (can be assigned manually to additional hosts), the rest will be available to the DHCP clients.
Every DHCP server will only be responsible for a part of the network. For example, the server on host1 will assign addresses from 172.16.42.10 to 172.16.42.19, whereas host2 will will assign addresses from172.16.42.20 to 172.16.42.29.
Note
This is just an example — you can expand the default values to run more than 10 containers on one host.
The host1 docker0 network interface will have the 172.16.42.1 address assigned, host2 will have172.16.42.2, and so on.

Make it work!

I assume that we have already started host1 and host2 as explained above.

Networking

Now it’s time to prepare the networking on both hosts. Log-in to both hosts and get the IP addresses of theeth0 network interfaces. Now run the network.sh script (it’s located in the /home/fedora directory) on both hosts.
Note
I assume that host1 has 192.168.122.31 IP and host2 has 192.168.122.2.
On host1 run the script with the IP of host2:
$ sudo ./network.sh 1 192.168.122.2
And do the opposite on host2:
$ sudo ./network.sh 2 192.168.122.31
Note
When you run the network.sh script for the first time, you may see messages similar tobridge docker0 doesn’t exist — don’t worry, this is normal.
The GRE tunnel should now be established and a DHCP server should be running on host1. You can confirm this by pinging the docker0 bridge addresses on each host.

Containers

There is one requirement for the container image — it needs to have a DHCP client installed. Sadly the defaultfedora image does not have the dhclient package installed. To make things easy I prepared thegoldmann/fedora-dhcp image. The only difference between fedora image is the addition of dhclient.
Download this image on both hosts:
docker pull goldmann/fedora-dhcp
If you run the goldmann/fedora-dhcp image you’ll see that there is no network interfaces beside the loopback. This is because Docker is run with the -b=none flag and it does not know about any network interfaces to bind to, so it does not create the ethernet adapter in the container.
But we still want to have networking. The only option at the moment is to use the -lxc-conf flag when running the image, like this:
docker run -i -t \
-lxc-conf="lxc.network.type = veth" \
-lxc-conf="lxc.network.link = docker0" \
-lxc-conf="lxc.network.flags = up" \
goldmann/fedora-dhcp /bin/bash
This will start a new container with a virtual ethernet adapter which is attached to the docker0 bridge. Sweet!

Obtaining the IP address

Since the Docker container does not run anything besides the command you specify (in our case /bin/bash) — it does not run the scripts that configures the network too. We need to do it by hand.
Note
I hope this will change in the near future. One option is to make systemd run well in the Docker containers.
After you get the prompt from the container, you can simply run the dhclient command. This will obtain the address from the DHCP server, exit and leave a shell just for you.
bash-4.2# ip a s dev eth0
17: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 1e:d4:13:c7:9d:fd brd ff:ff:ff:ff:ff:ff
    inet6 fe80::1cd4:13ff:fec7:9dfd/64 scope link
       valid_lft forever preferred_lft forever
bash-4.2# dhclient
bash-4.2# ip a s dev eth0
17: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 1e:d4:13:c7:9d:fd brd ff:ff:ff:ff:ff:ff
    inet 172.16.42.14/24 brd 172.16.42.255 scope global dynamic eth0
       valid_lft 43197sec preferred_lft 43197sec
    inet6 fe80::1cd4:13ff:fec7:9dfd/64 scope link
       valid_lft forever preferred_lft forever
bash-4.2# ping -c 1 google.com
PING google.com (173.194.65.139) 56(84) bytes of data.
64 bytes from ee-in-f139.1e100.net (173.194.65.139): icmp_seq=1 ttl=39 time=55.6 ms

--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 55.672/55.672/55.672/0.000 ms
Note
You can also use the /etc/init.d/network restart command to obtain the IP.
You can (should!) try it on host1 and host2. You should get the same result with no IP conflict and be able to access the Internet as well as other containers on the network.
Enjoy!

Comments

Post a Comment

Popular posts from this blog

WMI Static Port configuration

Step one:  Set the DCOM config to use a static port.... Basically, run "dcomcnfg" from command prompt. Navigate the tree to My Computer > DCOM Config > Windows Management and Instrumentation, select properties of that folder. Go to the Endpoints tab Select Properties button for Connection-oriented TCP/IP Use static endpoint, set the port. Step 2:  Configure WMI to use a fixed port http://msdn.microsoft.com/en-us/library/bb219447(v=VS.85).aspx At the command prompt, type  winmgmt -standalonehost Stop the WMI service by typing the command  net stop "Windows Management Instrumentation" Restart the WMI service again in a new service host by typing  net start "Windows Management Instrumentation" Establish a new port number for the WMI service by typing  netsh firewall add portopening TCP 24158 WMIFixedPort Still testing this myself, so not 100% certain it works. Another way, use the Component Services Manager to set the ra
Read more

Optimizing your JVM for Best Performance

The better your JVM performs, the better your installation of Tomcat will perform. It's as simple as that. Getting the most out of your JVM is a matter of configuring its settings to match your real-world performance needs as closely as possible. Update your JVM to the latest version, establish some accurate benchmarks so you have a way of quantifying any changes you make, and then get down to business. Effective Memory Management The main thing to consider when tuning your JVM for Tomcat performance is how to avoid wasting  memory  and draining your server's power to process requests. Certain automatic JVM processes, such as garbage collection and memory reallocation, can chew through memory if they occur more frequently than necessary. You can make sure these processes only occur when they need to by using the JAVA_OPTS -Xmx and -Xms switches to control how JVM handles its heap memory. If your JVM is invoking garbage collection too frequently, use the -Xmx switch to s
Read more

How do I disable FOREIGN KEY checking for the time of database schema migration?

How do I disable  FOREIGN KEY  checking for the time of database schema migration? Currently I found engine-based commands like: PostgreSQL:  SET CONSTRAINTS ALL DEFERRED; MSSQL:  EXEC sp_msforeachtable "ALTER TABLE ? NOCHECK CONSTRAINT all"; MySQL:  SET FOREIGN_KEY_CHECKS = 0; HSQLDB:  SET REFERENTIAL_INTEGRITY FALSE; DB2:  SET INTEGRITY FOR table OFF; SQLite 3.7.16:  PRAGMA foreign_keys = false; Oracle 11g seems to have no such command at all Syntax significantly differ, in case of MSSQL I'm unable to recover previously disabled constraints (if any).
Read more