For this post I will be focusing on showing a few advanced configurations and cool things you can do with LXC. Ill show you how to add IPs to your containers so that you can get to them outside of the host, show you a couple different ways on how to deploy new containers and show you the safest way to incorporate LXC into production use by demonstrating unprivileged containers.
Adding IPs
Ensure that you have properly configured your hosts by bridging the interfaces (see Networking post) if you have not already done so. Remember that all of you containers operate from their own configuration file, /var/lib/lxc/container-name/config. This configuration file is where you will place all of the IP information. Open the file with your desired eidtor and add/fill in these parameters:
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.hwaddr = Y0:UR:MA:CA:DR:ES
lxc.network.name = eth0
lxc.network.flags = up
lxc.network.mtu = 1500
lxc.network.ipv4 = 192.168.0.150/23
lxc.network.ipv4.gateway = 192.168.0.1
lxc.network.flags = up
lxc.network.link = br0
lxc.network.hwaddr = Y0:UR:MA:CA:DR:ES
lxc.network.name = eth0
lxc.network.flags = up
lxc.network.mtu = 1500
lxc.network.ipv4 = 192.168.0.150/23
lxc.network.ipv4.gateway = 192.168.0.1
Correct networking parameters will allow for you to have multiple containers that are able to communicate with one another and also allow for you to ssh to them from anywhere on the same network. I would also advise to set these same configs on the containers interfaces using ifconfig or modifying the interface config files themselves. Please note that setting an IP in the config file doesn't set the IP on the containers interface. So for example if the containers interface is using dhcp but the config file contains a static IP, you will have 2 IP addresses that will actually respond to ping and ssh. You can set the static IP on one or the other or both to keep from having more than 1 IP.
It is also possible to extend networking capabilities of the container to enable it to reach outside of the network that it sits on (outside of the bridge). One example of this would be to enable your container to also reach the internet. You can use iptables from the host to route requests outside of the bridge to other adapters that are available on the host based on specific IP addresses or based on whole subnets. Below, eth1 is assumed to be the adapter that is capable of talking to other networks like the internet etc. Also be sure to add IPv4 forwarding to /etc/sysctl.conf.
# iptables -t nat -A POSTROUTING -s 192.168.0.150 -o eth1 -j MASQUERADE
Or based on subnet.
# iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -o eth1 -j MASQUERADE
Add following line in /etc/systctl.conf to add IPv4 forwarding followed by issuing "systctl -p" to take effect without reboot.
net.ipv4.ip_forward = 1
# systctl -p
NOTE: If using Red Hat 7/CentOS 7 you will need to check firewalld settings. I completely disabled firewalld and continued using iptables instead since I am more familiar with iptables.
Cloning Containers
Another cool feature of LXC is the ability to clone individual containers as you would a virtual machine. Cloning provides the capability of faster deployments for custom configurations and I would also think it would provide a way for developers to auto scale if needed. There are 2 types of clones: Copy and Snapshots.From lxc-clone manpage: "A copy clone copies the root filessytem from the original container to the new. A snapshot filesystem uses the backing store's snapshot functionality to create a very small copy-on-write snapshot of the original container. Snapshot clones require the new container backing store to support snapshotting. Currently this includes only aufs, btrfs, lvm, overlayfs and zfs. LVM devices do not support snapshots of snapshots."
For purposes of this post we will keep it simple and create a simple copy type clone. The "-o" flag defines original container name that will be cloned and the "-n" flag defines new container name. From my experience the rootfs, hostname and the mac will not duplicate but I haven't found a way to keep the IP from duplicating. So just remember if you are using a static IP that you will need to update the IP info on the new container to keep from conflicting. There is probably a way to script it but by default there is no way around it that I have found yet.
# lxc-clone -o CONTAINER_TOBE_CLONED -n NEW_CONTAINER
Should get an output like "Created container NEW_CONTAINER as copy of fedora-master" when clone is completed.
Autostart
This is one of the coolest features of LXC in my opinion. Since each container is run as a process, each process (container) has the ability of being started at bootup. The LXC program can be started/run as a service (service lxc start | systemctl start lxc) and when this service is brought to life it can tell specfific containers to startup with the LXC service. This is done through passing the autostart parameter(s) to the containers config file. The autostart options support marking which containers should be auto-started and in what order and can be based on either a number or a group. See man page for lxc.container.conf for extended details.
For the most basic autostart option, pass the following parameter to the contianers config file:
lxc.start.auto = 1 # 0 value means off. 1 value means on.
Other parameters worth mentioning for autostart are the start order parameter and start delay. The start order along with the start delay can help bring up containers in a certain order and set a wait time before starting the next one. Can be helpful for multi container environments such as a database.
lxc.start.order = N # where N is a number
lxc.start.delay = N # where N is a number in seconds
For the most basic autostart option, pass the following parameter to the contianers config file:
lxc.start.auto = 1 # 0 value means off. 1 value means on.
Other parameters worth mentioning for autostart are the start order parameter and start delay. The start order along with the start delay can help bring up containers in a certain order and set a wait time before starting the next one. Can be helpful for multi container environments such as a database.
lxc.start.order = N # where N is a number
lxc.start.delay = N # where N is a number in seconds
Unprivileged Containers
Unprivileged containers are perhaps the safest way to deploy LXC especially in a production environment. LXC gets a bad rap for being unsecure at times and has actually allowed users to gain access to the root account on the host. This is possible because although containers all run in a separate namespace, uid 0 in your container is still equal to uid 0 outside of the container. Unprivileged containers run as a non root process on the host even though they can have root inside of the container itself. So at a high level we need to remap these namespaces and ensure that these processes are not running under root's uid. A little confusing but hopefully after you see a demo it will make sense. Stephane Graber who is one of the lead developers behind LXC sums this up really well in his blog on unprivileged containers.
"So how do those user namespaces work? Well, simply put, each user that’s allowed to use them on the system gets assigned a range of unused uids and gids, ideally a whole 65536 of them. You can then use those uids and gids with two standard tools called newuidmap and newgidmap which will let you map any of those uids and gids to virtual uids and gids in a user namespace." - Stephane Graber "LXC 1.0: Unprivileged Containers"
With the development of unprivileged containers we are able to allow users other than root to start an container although with unprivileged containers there are still limits on some things a user can do in namespace. I will demonstrate this on a Ubuntuxc 14.04 since my CentOS 7 box doesnt meet the prereq kernel features needed for this. The tools that we will need to configure this (newuidmap and newgidmap) require kernel verison 3.12 or higher. Let's Begin!
Ensure that you have uidmap package installed:
# sudo apt-get install uidmap
Then assign your user subuids and subgids and give execution privileges to user home.
# sudo usermod --add-subuids 100000-165536 USER
# sudo usermod --add-subgids 100000-165536 USER
# sudo chmod +x /home/USER
Add the mappings as part of the container parameters in ~/.config/lxc/default.
lxc.id_map = u 0 100000 65536
lxc.id_map = g 0 100000 65536
Create and start the unprivileged container. Note this will take some time to complete.
# lxc-create -t download -n ubuntu-unprived -- -d ubuntu -r trusty -a amd64
# lxc-start -n ubuntu-unprived -d
So now lets compare what the processes look like from the host and from the container for this namespace.
From the container:
# lxc-attach -n ubuntu-unprived
root@ubuntu-unprived:/# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 04:48 ? 00:00:00 /sbin/init
root 157 1 0 04:48 ? 00:00:00 upstart-udev-bridge --daemon
root 189 1 0 04:48 ? 00:00:00 /lib/systemd/systemd-udevd --daemon
root 244 1 0 04:48 ? 00:00:00 dhclient -1 -v -pf /run/dhclient.eth0.pid
syslog 290 1 0 04:48 ? 00:00:00 rsyslogd
root 343 1 0 04:48 tty4 00:00:00 /sbin/getty -8 38400 tty4
root 345 1 0 04:48 tty2 00:00:00 /sbin/getty -8 38400 tty2
root 346 1 0 04:48 tty3 00:00:00 /sbin/getty -8 38400 tty3
root 359 1 0 04:48 ? 00:00:00 cron
root 386 1 0 04:48 console 00:00:00 /sbin/getty -8 38400 console
root 389 1 0 04:48 tty1 00:00:00 /sbin/getty -8 38400 tty1
root 408 1 0 04:48 ? 00:00:00 upstart-socket-bridge --daemon
root 409 1 0 04:48 ? 00:00:00 upstart-file-bridge --daemon
root 431 0 0 05:06 ? 00:00:00 /bin/bash
root 434 431 0 05:06 ? 00:00:00 ps -ef
From the host:
# lxc-info -Ssip --name ubuntu-unprived
State: RUNNING
PID: 3104
IP: 10.1.0.107
CPU use: 2.27 seconds
BlkIO use: 680.00 KiB
Memory use: 7.24 MiB
Link: vethJ1Y7TG
TX bytes: 7.30 KiB
RX bytes: 46.21 KiB
Total bytes: 53.51 KiB
# ps -ef | grep 3104
100000 3104 3067 0 Nov11 ? 00:00:00 /sbin/init
100000 3330 3104 0 Nov11 ? 00:00:00 upstart-udev-bridge --daemon
100000 3362 3104 0 Nov11 ? 00:00:00 /lib/systemd/systemd-udevd --daemon
100000 3417 3104 0 Nov11 ? 00:00:00 dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
100102 3463 3104 0 Nov11 ? 00:00:00 rsyslogd
100000 3516 3104 0 Nov11 pts/8 00:00:00 /sbin/getty -8 38400 tty4
100000 3518 3104 0 Nov11 pts/6 00:00:00 /sbin/getty -8 38400 tty2
100000 3519 3104 0 Nov11 pts/7 00:00:00 /sbin/getty -8 38400 tty3
100000 3532 3104 0 Nov11 ? 00:00:00 cron
100000 3559 3104 0 Nov11 pts/9 00:00:00 /sbin/getty -8 38400 console
100000 3562 3104 0 Nov11 pts/5 00:00:00 /sbin/getty -8 38400 tty1
100000 3581 3104 0 Nov11 ? 00:00:00 upstart-socket-bridge --daemon
100000 3582 3104 0 Nov11 ? 00:00:00 upstart-file-bridge --daemon
lxc 3780 1518 0 00:10 pts/4 00:00:00 grep --color=auto 3104
As you can see processes are running inside the container as root but are not appearing as root but as 100000 from the host.
Hope you enjoyed this blog post. Next in the series will begin talking about the ever popular Docker and ease your into some really cool projects going on around it.
Blog Series on Linux Containers:
Previous Post: First Container
Next Post: Introduction to Docker
"So how do those user namespaces work? Well, simply put, each user that’s allowed to use them on the system gets assigned a range of unused uids and gids, ideally a whole 65536 of them. You can then use those uids and gids with two standard tools called newuidmap and newgidmap which will let you map any of those uids and gids to virtual uids and gids in a user namespace." - Stephane Graber "LXC 1.0: Unprivileged Containers"
With the development of unprivileged containers we are able to allow users other than root to start an container although with unprivileged containers there are still limits on some things a user can do in namespace. I will demonstrate this on a Ubuntuxc 14.04 since my CentOS 7 box doesnt meet the prereq kernel features needed for this. The tools that we will need to configure this (newuidmap and newgidmap) require kernel verison 3.12 or higher. Let's Begin!
Ensure that you have uidmap package installed:
# sudo apt-get install uidmap
Then assign your user subuids and subgids and give execution privileges to user home.
# sudo usermod --add-subuids 100000-165536 USER
# sudo usermod --add-subgids 100000-165536 USER
# sudo chmod +x /home/USER
Add the mappings as part of the container parameters in ~/.config/lxc/default.
lxc.id_map = u 0 100000 65536
lxc.id_map = g 0 100000 65536
# lxc-create -t download -n ubuntu-unprived -- -d ubuntu -r trusty -a amd64
# lxc-start -n ubuntu-unprived -d
So now lets compare what the processes look like from the host and from the container for this namespace.
From the container:
# lxc-attach -n ubuntu-unprived
root@ubuntu-unprived:/# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 04:48 ? 00:00:00 /sbin/init
root 157 1 0 04:48 ? 00:00:00 upstart-udev-bridge --daemon
root 189 1 0 04:48 ? 00:00:00 /lib/systemd/systemd-udevd --daemon
root 244 1 0 04:48 ? 00:00:00 dhclient -1 -v -pf /run/dhclient.eth0.pid
syslog 290 1 0 04:48 ? 00:00:00 rsyslogd
root 343 1 0 04:48 tty4 00:00:00 /sbin/getty -8 38400 tty4
root 345 1 0 04:48 tty2 00:00:00 /sbin/getty -8 38400 tty2
root 346 1 0 04:48 tty3 00:00:00 /sbin/getty -8 38400 tty3
root 359 1 0 04:48 ? 00:00:00 cron
root 386 1 0 04:48 console 00:00:00 /sbin/getty -8 38400 console
root 389 1 0 04:48 tty1 00:00:00 /sbin/getty -8 38400 tty1
root 408 1 0 04:48 ? 00:00:00 upstart-socket-bridge --daemon
root 409 1 0 04:48 ? 00:00:00 upstart-file-bridge --daemon
root 431 0 0 05:06 ? 00:00:00 /bin/bash
root 434 431 0 05:06 ? 00:00:00 ps -ef
From the host:
# lxc-info -Ssip --name ubuntu-unprived
State: RUNNING
PID: 3104
IP: 10.1.0.107
CPU use: 2.27 seconds
BlkIO use: 680.00 KiB
Memory use: 7.24 MiB
Link: vethJ1Y7TG
TX bytes: 7.30 KiB
RX bytes: 46.21 KiB
Total bytes: 53.51 KiB
# ps -ef | grep 3104
100000 3104 3067 0 Nov11 ? 00:00:00 /sbin/init
100000 3330 3104 0 Nov11 ? 00:00:00 upstart-udev-bridge --daemon
100000 3362 3104 0 Nov11 ? 00:00:00 /lib/systemd/systemd-udevd --daemon
100000 3417 3104 0 Nov11 ? 00:00:00 dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
100102 3463 3104 0 Nov11 ? 00:00:00 rsyslogd
100000 3516 3104 0 Nov11 pts/8 00:00:00 /sbin/getty -8 38400 tty4
100000 3518 3104 0 Nov11 pts/6 00:00:00 /sbin/getty -8 38400 tty2
100000 3519 3104 0 Nov11 pts/7 00:00:00 /sbin/getty -8 38400 tty3
100000 3532 3104 0 Nov11 ? 00:00:00 cron
100000 3559 3104 0 Nov11 pts/9 00:00:00 /sbin/getty -8 38400 console
100000 3562 3104 0 Nov11 pts/5 00:00:00 /sbin/getty -8 38400 tty1
100000 3581 3104 0 Nov11 ? 00:00:00 upstart-socket-bridge --daemon
100000 3582 3104 0 Nov11 ? 00:00:00 upstart-file-bridge --daemon
lxc 3780 1518 0 00:10 pts/4 00:00:00 grep --color=auto 3104
Hope you enjoyed this blog post. Next in the series will begin talking about the ever popular Docker and ease your into some really cool projects going on around it.
Blog Series on Linux Containers:
Previous Post: First Container
Next Post: Introduction to Docker
