Ultimate Container Sandbox | Isolating Containers in Containers

This was something fun I worked on for while to display how to give users a safe development box to do things like learn, play or test with docker. Its an extremely ephemeral environment and can be rebuilt in secs. It has been sitting in my drafts for a bit but wanted to write about it...... 

Anyone that has been involved in the docker ecosystem over the past several years has more than likely seen the following image below:

Running docker inside of docker. This is nothing new and in fact if you are using Docker universally to run virtually everything such as monitoring or service discovery, chances are you are most likely mounting the docker socket inside your container. I personally use docker in docker to build and push doing the same thing. 

This is where it gets hairy and you get into the inception aspect of this whole mess.

The cool thing with running docker in docker is the fact that you are able to give yourself a nice little test bed with no worries of destroying ready containers and also utilize docker command line at the same time. Building and push new images etc. The only issue with this is the fact that you are mounting the docker socket within the container itself. You are exposing the hosts images and containers to the docker in docker. If you run a 'docker images' inside the docker container, you are seeing the hosts images. If you run a 'docker rm|rmi' you will wipe the host you are running on. There is NO isolation in this. Not only would you wipe the host but anyone else that is running docker in docker on the host would be doing the same thing. 

One way I have figured out how to isolate docker running on the same host is to utilize docker's father project, LXC. By running docker inside of LXC, each LXC instance is completely isolated from the other and you are safely able to utilize docker without affecting anyone else. As with docker, LXC can also be spun up in a matter of seconds so in the event that you do something in LXC that you dont like, blow it away and spin up a new. Good read and another instance of this being used: Openstack Carina Project

Docker on LXC on Linux

Image provide by yours truly... You're Welcome!

Let us get this going. Ubuntu as the underlying host OS as I am starting to go back to my original Linux roots.

1) Install LXC:

# apt-get update && apt-get install lxc

2) Create the LXC container and add the following lines to each containers configs /var/lib/lxc/$LXC_NAME/config:
# lxc-create -t download -n meh-01 -- -d ubuntu -r trusty -a amd64      
  

Add below lines to /var/lib/lxc/meh-01/config  

lxc.aa_profile = unconfined

lxc.cgroup.devices.allow = a
lxc.cap.drop =

3) Start the LXC container, attach and install the needful to get docker installed in LXC:
# lxc-start -n meh-01 -d 
# lxc-attach -n meh-01

Inside LXC:
# apt-get update && apt-get install wget apparmor docker.io -y

4) Check it out!!! 

FROM LXC:
root@meh-01:~# docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.2.1
Git commit (client): 7c8fca2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.2.1
Git commit (server): 7c8fca2
OS/Arch (server): linux/amd64

root@meh-01:~# docker images
REPOSITORY                       TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
golang                           latest              471e087e791d        2 weeks ago         744 MB

root@meh-01:~# docker run -it golang echo hello world
hello world

FROM HOST:
root@docker-builder:~# docker images
The program 'docker' is currently not installed. You can install it by typing:
apt-get install docker

Docker isn't even installed on the host so the host is not being affected... ***Modify changes to your docker options within LXC if you would like to add things like private registry etc...

Next: Create another LXC container and repeat the above steps and notice you get complete isolation and separate development environments with LXC. Add things into the LXC containers such as ssh and port forwarding on the host so you can SSH to it. 

LXC is the original container runtime that got me interested in containers (my blog from a couple years ago). I will continue to use alongside docker for different things because I think that LXC has some functionality the docker doesn't do as well. For example, running OS containers, LXC is much better. Docker still holds the belt for application containers in my opinion. Be sure to check out Rackspace's CaaS mentioned above. Awesome project and read. I will be following not only what they are doing but Openstack as well. 

CONTAINERIZE ALL THE THINGS