Tuesday, November 11, 2014

The Container World | Part 5 Advanced Configuration

For this post I will be focusing on showing a few advanced configurations and cool things you can do with LXC. Ill show you how to add IPs to your containers so that you can get to them outside of the host, show you a couple different ways on how to deploy new containers and show you the safest way to incorporate LXC into production use by demonstrating unprivileged containers.

Adding IPs

Ensure that you have properly configured your hosts by bridging the interfaces (see Networking post) if you have not already done so. Remember that all of you containers operate from their own configuration file, /var/lib/lxc/container-name/config. This configuration file is where you will place all of the IP information. Open the file with your desired eidtor and add/fill in these parameters:

    lxc.network.type = veth
    lxc.network.flags = up
    lxc.network.link = br0
    lxc.network.hwaddr = Y0:UR:MA:CA:DR:ES
    lxc.network.name = eth0
    lxc.network.flags = up
    lxc.network.mtu = 1500
    lxc.network.ipv4 =
    lxc.network.ipv4.gateway =

Correct networking parameters will allow for you to have multiple containers that are able to communicate with one another and also allow for you to ssh to them from anywhere on the same network. I would also advise to set these same configs on the containers interfaces using ifconfig or modifying the interface config files themselves. Please note that setting an IP in the config file doesn't set the IP on the containers interface. So for example if the containers interface is using dhcp but the config file contains a static IP, you will have 2 IP addresses that will actually respond to ping and ssh. You can set the static IP on one or the other or both to keep from having more than 1 IP.

It is also possible to extend networking capabilities of the container to enable it to reach outside of the network that it sits on (outside of the bridge). One example of this would be to enable your container to also reach the internet. You can use iptables from the host to route requests outside of the bridge to other adapters that are available on the host based on specific IP addresses or based on whole subnets. Below, eth1 is assumed to be the adapter that is capable of talking to other networks like the internet etc. Also be sure to add IPv4 forwarding to /etc/sysctl.conf.

    # iptables -t nat -A POSTROUTING -s -o eth1 -j MASQUERADE

Or based on subnet.

    # iptables -t nat -A POSTROUTING -s -o eth1 -j MASQUERADE

Add following line in /etc/systctl.conf to add IPv4 forwarding followed by issuing "systctl -p" to take effect without reboot.

    net.ipv4.ip_forward = 1

    # systctl -p

NOTE: If using Red Hat 7/CentOS 7 you will need to check firewalld settings. I completely disabled firewalld and continued using iptables instead since I am more familiar with iptables.


Cloning Containers

Another cool feature of LXC is the ability to clone individual containers as you would a virtual machine. Cloning provides the capability of faster deployments for custom configurations and I would also think it would provide a way for developers to auto scale if needed. There are 2 types of clones: Copy and Snapshots. 

From lxc-clone manpage: "A copy clone copies the root filessytem from the original  container  to  the  new.  A snapshot  filesystem uses the backing store's snapshot functionality to create a very small copy-on-write snapshot of the original container. Snapshot clones require the new container backing  store  to  support snapshotting.  Currently  this includes only aufs, btrfs, lvm, overlayfs and zfs. LVM devices do not support snapshots of snapshots."    

For purposes of this post we will keep it simple and create a simple copy type clone. The "-o" flag defines original container name that will be cloned and the "-n" flag defines new container name. From my experience the rootfs, hostname and the mac will not duplicate but I haven't found a way to keep the IP from duplicating. So just remember if you are using a static IP that you will need to update the IP info on the new container to keep from conflicting. There is probably a way to script it but by default there is no way around it that I have found yet.


Should get an output like "Created container NEW_CONTAINER as copy of fedora-master" when clone is completed.


This is one of the coolest features of LXC in my opinion. Since each container is run as a process, each process (container) has the ability of being started at bootup. The LXC program can be started/run as a service (service lxc start | systemctl start lxc) and when this service is brought to life it can tell specfific containers to startup with the LXC service. This is done through passing the autostart parameter(s) to the containers config file. The autostart options support marking which containers should be auto-started and in what order and can be based on either a number or a groupSee man page for lxc.container.conf for extended details.

For the most basic autostart option, pass the following parameter to the contianers config file:

    lxc.start.auto = 1   # 0 value means off. 1 value means on.

Other parameters worth mentioning for autostart are the start order parameter and start delay. The start order along with the start delay can help bring up containers in a certain order and set a wait time before starting the next one. Can be helpful for multi container environments such as a database.

    lxc.start.order = N  # where N is a number
    lxc.start.delay = N  # where N is a number in seconds

Unprivileged Containers

Unprivileged containers are perhaps the safest way to deploy LXC especially in a production environment. LXC gets a bad rap for being unsecure at times and has actually allowed users to gain access to the root account on the host. This is possible because although containers all run in a separate namespace, uid 0 in your container is still equal to uid 0 outside of the container. Unprivileged containers run as a non root process on the host even though they can have root inside of the container itself. So at a high level we need to remap these namespaces and ensure that these processes are not running under root's uid. A little confusing but hopefully after you see a demo it will make sense. Stephane Graber who is one of the lead developers behind LXC sums this up really well in his blog on unprivileged containers.

"So how do those user namespaces work? Well, simply put, each user that’s allowed to use them on the system gets assigned a range of unused uids and gids, ideally a whole 65536 of them. You can then use those uids and gids with two standard tools called newuidmap and newgidmap which will let you map any of those uids and gids to virtual uids and gids in a user namespace." - Stephane Graber "LXC 1.0: Unprivileged Containers"

With the development of unprivileged containers we are able to allow users other than root to start an container although with unprivileged containers there are still limits on some things a user can do in namespace. I will demonstrate this on a Ubuntuxc 14.04 since my CentOS 7 box doesnt meet the prereq kernel features needed for this. The tools that we will need to configure this (newuidmap and newgidmap) require kernel verison 3.12 or higher. Let's Begin!

Ensure that you have uidmap package installed:

    # sudo apt-get install uidmap

Then assign your user subuids and subgids and give execution privileges to user home. 

    # sudo usermod --add-subuids 100000-165536 USER
    # sudo usermod --add-subgids 100000-165536 USER
    # sudo chmod +x /home/USER

Add the mappings as part of the container parameters in ~/.config/lxc/default.

    lxc.id_map = u 0 100000 65536

    lxc.id_map = g 0 100000 65536

Create and start the unprivileged container. Note this will take some time to complete.

    # lxc-create -t download -n ubuntu-unprived -- -d ubuntu -r trusty -a amd64
    # lxc-start -n ubuntu-unprived -d

So now lets compare what the processes look like from the host and from the container for this namespace.

From the container:
# lxc-attach -n ubuntu-unprived

root@ubuntu-unprived:/# ps -ef
root         1     0  0 04:48 ?        00:00:00 /sbin/init
root       157     1  0 04:48 ?        00:00:00 upstart-udev-bridge --daemon
root       189     1  0 04:48 ?        00:00:00 /lib/systemd/systemd-udevd --daemon
root       244     1  0 04:48 ?        00:00:00 dhclient -1 -v -pf /run/dhclient.eth0.pid
syslog     290     1  0 04:48 ?        00:00:00 rsyslogd
root       343     1  0 04:48 tty4     00:00:00 /sbin/getty -8 38400 tty4
root       345     1  0 04:48 tty2     00:00:00 /sbin/getty -8 38400 tty2
root       346     1  0 04:48 tty3     00:00:00 /sbin/getty -8 38400 tty3
root       359     1  0 04:48 ?        00:00:00 cron
root       386     1  0 04:48 console  00:00:00 /sbin/getty -8 38400 console
root       389     1  0 04:48 tty1     00:00:00 /sbin/getty -8 38400 tty1
root       408     1  0 04:48 ?        00:00:00 upstart-socket-bridge --daemon
root       409     1  0 04:48 ?        00:00:00 upstart-file-bridge --daemon
root       431     0  0 05:06 ?        00:00:00 /bin/bash

root       434   431  0 05:06 ?        00:00:00 ps -ef

From the host:

# lxc-info -Ssip --name ubuntu-unprived
State:          RUNNING
PID:            3104
CPU use:        2.27 seconds
BlkIO use:      680.00 KiB
Memory use:     7.24 MiB
Link:           vethJ1Y7TG
 TX bytes:      7.30 KiB
 RX bytes:      46.21 KiB
 Total bytes:   53.51 KiB

# ps -ef | grep 3104
100000    3104  3067  0 Nov11 ?        00:00:00 /sbin/init
100000    3330  3104  0 Nov11 ?        00:00:00 upstart-udev-bridge --daemon
100000    3362  3104  0 Nov11 ?        00:00:00 /lib/systemd/systemd-udevd --daemon
100000    3417  3104  0 Nov11 ?        00:00:00 dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
100102    3463  3104  0 Nov11 ?        00:00:00 rsyslogd
100000    3516  3104  0 Nov11 pts/8    00:00:00 /sbin/getty -8 38400 tty4
100000    3518  3104  0 Nov11 pts/6    00:00:00 /sbin/getty -8 38400 tty2
100000    3519  3104  0 Nov11 pts/7    00:00:00 /sbin/getty -8 38400 tty3
100000    3532  3104  0 Nov11 ?        00:00:00 cron
100000    3559  3104  0 Nov11 pts/9    00:00:00 /sbin/getty -8 38400 console
100000    3562  3104  0 Nov11 pts/5    00:00:00 /sbin/getty -8 38400 tty1
100000    3581  3104  0 Nov11 ?        00:00:00 upstart-socket-bridge --daemon
100000    3582  3104  0 Nov11 ?        00:00:00 upstart-file-bridge --daemon
lxc       3780  1518  0 00:10 pts/4    00:00:00 grep --color=auto 3104

As you can see processes are running inside the container as root but are not appearing as root but as 100000 from the host.

Hope you enjoyed this blog post. Next in the series will begin talking about the ever popular Docker and ease your into some really cool projects going on around it.

Blog Series on Linux Containers:
Previous Post: First Container
Next Post: Introduction to Docker

1 comment:

  1. To install the latest kernel on centos see:


    (currently the 3.19 kernel on Centos 7)