Fedora Atomic, OpenStack, and Kubernetes

While experimenting with Fedora Atomic, I was looking for an elegant way to automatically deploy Atomic into anOpenStack environment and then automatically schedule some Docker containers on the Atomic host. This post describes my solution.
Like many other cloud-targeted distributions, Fedora Atomic runs cloud-init when the system boots. We can take advantage of this to configure the system at first boot by providing a user-data blob to Nova when we boot the instance. A user-data blob can be as simple as a shell script, and while we could arguably mash everything into a single script it wouldn't be particularly maintainable or flexible in the face of different pod/service/etc descriptions.
In order to build a more flexible solution, we're going to take advantage of the following features:
  • Cloud-init allows you to pass in multiple files via user-data by encoding them as a multipart MIME archive.
  • Support for a custom part handler.
    Cloud-init recognizes a number of specific MIME types (such as text/cloud-config or text/x-shellscript). We can provide a custom part handler that will be used to handle MIME types not intrinsincally supported by cloud-init.

A custom part handler for Kubernetes configurations

I have written a custom part handler that knows about the following MIME types:
  • text/x-kube-pod
  • text/x-kube-service
  • text/x-kube-replica
When the part handler is first initialized it will ensure the Kubernetes is started. If it is provided with a document matching one of the above MIME types, it will pass it to the appropriate kubecfg command to create the objects in Kubernetes.

Creating multipart MIME archives

I have also created a modified version of the standard write-multipart-mime.py Python script. This script will inspect the first lines of files to determine their content type; in addition to the standard cloud-init types (like #cloud-config for a text/cloud-config type file), this script recognizes:
  • #kube-pod for text/x-kube-pod
  • #kube-service for text/x-kube-service
  • #kube-replica for text/x-kube-replca
That is, a simple pod description might look something like:
#kube-pod
id: dbserver
desiredState:
  manifest:
    version: v1beta1
    id: dbserver
    containers:
    - image: mysql
      name: dbserver
      env:
        - name: MYSQL_ROOT_PASSWORD
          value: secret

Putting it all together

Assuming that the pod description presented in the previous section is stored in a file named dbserver.yaml, we can bundle that file up with our custom part handler like this:
$ write-mime-multipart.py \
  kube-part-handler.py dbserver.yaml > userdata
We would then launch a Nova instance using the nova boot command, providing the generated userdata file as an argument to the user-data command:
$ nova boot --image fedora-atomic --key-name mykey \
  --flavor m1.small --user-data userdata my-atomic-server
You would obviously need to substitute values for --image and --key-name that are appropriate for your environment.

Details, details

If you are experimenting with Fedora Atomic 21, you may find out that the above example doesn't work -- the official mysql image generates an selinux error. We can switch selinux to permissive mode by putting the following into a file called disable-selinux.sh:
#!/bin/sh

setenforce 0
sed -i '/^SELINUX=/ s/=.*/=permissive/' /etc/selinux/config
And then including that in our MIME archive:
$ write-mime-multipart.py \
  kube-part-handler.py disable-selinux.sh dbserver.yaml > userdata

A brief demonstration

If we launch an instance as described in the previous section and then log in, we should find that the pod has already been scheduled:
# kubecfg list pods
ID                  Image(s)            Host                Labels              Status
----------          ----------          ----------          ----------          ----------
dbserver            mysql               /                                       Waiting
At this point, docker needs to pull the mysql image locally, so this step can take a bit depending on the state of your local internet connection.
Running docker ps at this point will yield:
# docker ps
CONTAINER ID        IMAGE                     COMMAND             CREATED             STATUS              PORTS               NAMES
3561e39f198c        kubernetes/pause:latest   "/pause"            46 seconds ago      Up 43 seconds                           k8s--net.d96a64a9--dbserver.etcd--3d30eac0_-_745c_-_11e4_-_b32a_-_fa163e6e92ce--d872be51
The pause image here is a Kubernetes detail that is used to configure the networking for a pod (in the Kubernetes world, a pod is a group of linked containers that share a common network namespace).
After a few minutes, you should eventually see:
# docker ps
CONTAINER ID        IMAGE                     COMMAND                CREATED             STATUS              PORTS               NAMES
644c8fc5a79c        mysql:latest              "/entrypoint.sh mysq   3 minutes ago       Up 3 minutes                            k8s--dbserver.fd48803d--dbserver.etcd--3d30eac0_-_745c_-_11e4_-_b32a_-_fa163e6e92ce--58794467   
3561e39f198c        kubernetes/pause:latest   "/pause"               5 minutes ago       Up 5 minutes                            k8s--net.d96a64a9--dbserver.etcd--3d30eac0_-_745c_-_11e4_-_b32a_-_fa163e6e92ce--d872be51
And kubecfg should show the pod as running:
# kubecfg list pods
ID                  Image(s)            Host                Labels              Status
----------          ----------          ----------          ----------          ----------
dbserver            mysql               127.0.0.1/                              Running

Problems, problems

This works and is I think a relatively elegant solution. However, there are some drawbacks. In particular, the custom part handler runs fairly early in the cloud-init process, which means that it cannot depend on changes implemented by user-data scripts (because these run much later).
A better solution might be to have the custom part handler simply write the Kubernetes configs into a directory somewhere, and then install a service that launches after Kubernetes and (a) watches that directory for files, then (b) passes the configuration to Kubernetes and deletes (or relocates) the file.

Comments

Post a Comment

Popular posts from this blog

WMI Static Port configuration

Step one:  Set the DCOM config to use a static port.... Basically, run "dcomcnfg" from command prompt. Navigate the tree to My Computer > DCOM Config > Windows Management and Instrumentation, select properties of that folder. Go to the Endpoints tab Select Properties button for Connection-oriented TCP/IP Use static endpoint, set the port. Step 2:  Configure WMI to use a fixed port http://msdn.microsoft.com/en-us/library/bb219447(v=VS.85).aspx At the command prompt, type  winmgmt -standalonehost Stop the WMI service by typing the command  net stop "Windows Management Instrumentation" Restart the WMI service again in a new service host by typing  net start "Windows Management Instrumentation" Establish a new port number for the WMI service by typing  netsh firewall add portopening TCP 24158 WMIFixedPort Still testing this myself, so not 100% certain it works. Another way, use the Component Services Manager to set the ra...
Read more

Optimizing your JVM for Best Performance

The better your JVM performs, the better your installation of Tomcat will perform. It's as simple as that. Getting the most out of your JVM is a matter of configuring its settings to match your real-world performance needs as closely as possible. Update your JVM to the latest version, establish some accurate benchmarks so you have a way of quantifying any changes you make, and then get down to business. Effective Memory Management The main thing to consider when tuning your JVM for Tomcat performance is how to avoid wasting  memory  and draining your server's power to process requests. Certain automatic JVM processes, such as garbage collection and memory reallocation, can chew through memory if they occur more frequently than necessary. You can make sure these processes only occur when they need to by using the JAVA_OPTS -Xmx and -Xms switches to control how JVM handles its heap memory. If your JVM is invoking garbage collection too frequently, use the -Xmx switch to ...
Read more

Exploring LXC Networking

Recently I’ve been finding myself in various conversations about  Docker  and  Linux Containers (LXC) . Most of the time the conversations eventually end up with one and the same question and that is whether we should run containers in production. Initially this post had a few paragraphs where I philosophised about readiness of the technology, but I’ve deleted those paragraphs now as the attention dedicated to containers in past year has been nothing short of amazing and more and more companies are running their Infrastructures in the containers or at least big parts of them. I’m sure this trend will continue to high degree in the future, so I’ve now removed these paragraphs and kept the technical content only. One of the (many) things which were not entirely clear to me and to the people I speak with about the topic of containers almost on daily basis is how the networking can be done and configured when using LXC. Hopefully the first blog post on this topic is going...
Read more