Friday, May 13, 2016

Orchestrating Communications to Docker as you Scale Like a Boss

One of the more difficult things to manage as you begin to scale and deploy containers at mass is trying to manage communications and access to your services. Not only is managing communications to your services difficult, but also doing it in a way that makes sense, is static and feels normal for your users. With an orchestration tool such as Mesos, your containers will most likely move from host to host quite often. This is exactly how it should be for environments running large amounts of containers. It shouldn't matter where your container lives and you should also not have to search for it as it moves around. Nor should you have to manage ports as your Infrastructure grows and your apps scale. I believe this to be one of the major pieces to consider when planning your container based environment. Think about the reason that you are considering using containers and then think about how you plan to orchestrate access to them and assume that your services will not be running in the same place tomorrow as they were today. We achieve this solution through a simple mechanism of Service Discovery and Load Balancing.

In this post, Ill describe the tools that I have chosen to use in my docker based PaaS solution backed by Apache Mesos. I went with a solution that not only would suit the needs of our Mesos based services but would work along side any docker container that was deployed in the environment. Simply setup a node with load balancing and access to your service discovery and have the users route through this node to access their service.

Demonstrations will be done using Marathon, Consul, consul-template + HaProxy, but as I said there are a ton of projects out there that can be used to help solve this issue. 

Components used:


Workflow:
  1. Docker service deployed with Marathon to Mesos
  2. Registrator running on Mesos Agents registers the service to Consul
  3. consul-template updates HAProxy with port mappings of service(s) and reloads config
  4. ACCESS TO SERVICE(S)!!!!



































Getting Started. Note: You will need a running Mesos Cluster with Marathon and also a running Consul cluster. See my post for getting a Consul cluster up in 10 minutes here:

1) On a server that you would like to use to proxy traffic, install HAProxy and consul-template

# yum install -y haproxy unzip && cd /usr/local/bin/ && wget -O consul-template.zip wget https://releases.hashicorp.com/consul-template/0.14.0/consul-template_0.14.0_linux_amd64.zip
   
# unzip consul-template.zip 

2) Configure consul-template for HAProxy. It will reload the config each time there is a change with the service such as a scale up, down or a failure. 

# mkdir -pv /etc/consul-template/ && cd /etc/consul-template

Create new file /etc/consul-template/consul-haproxy.json which will be the configuration file to manage reloading haproxy anytime there is a change is service discovery. 

# cat /etc/consul-template/consul-haproxy.json
consul = "$CONSUL:$PORT"

template {
  source = "/etc/haproxy/haproxy.template"
  destination = "/etc/haproxy/haproxy.cfg"
  command = "systemctl reload haproxy"


}

Create the source and destination files for haproxy based on the config above.

# cat /etc/haproxy/haproxy.template
global
  daemon
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  maxconn 4096

defaults
  log            global
  retries             3
  maxconn          2000
  timeout connect  5000
  timeout client  50000
  timeout server  50000

listen http-in
  bind *:80
  mode tcp
  option tcplog
  balance leastconn{{range service "$SERVICE"}}
  server {{.Node}} {{.Address}}:{{.Port}} check {{end}}

$SERVICE in the template file above is the service name that you will put as part of a ENV parameter in your Marathon json when you launch. It will register itself in Consul as that service name and anytime there is a change if will reflect the change to haproxy.

# cat /etc/haproxy/haproxy.cfg
global
  daemon
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  maxconn 4096

defaults
  log            global
  retries             3
  maxconn          2000
  timeout connect  5000
  timeout client  50000
  timeout server  50000

listen http-in
  bind *:80
  mode tcp
  option tcplog
  balance leastconn 


3) We can go ahead and start consul-template at this point. Run from command line or from systemd unit to make it permanent.

# consul-template -config /etc/consul-template/consul-haproxy.json

OR

# cat /etc/systemd/system/consul-template.service
[Unit]
Description=Consul Template HA Proxy
After=network.target

[Service]
User=root
Group=root
Environment="GOMAXPROCS=2"
ExecStart=/usr/local/bin/consul-template -config /etc/consul-template/consul-haproxy.json
ExecReload=/bin/kill -9 $MAINPID
KillSignal=SIGINT
Restart=on-failure

[Install]
WantedBy=multi-user.target


# systemctl enable consul-template && systemctl start consul-template


4) Registrator must be running on any host in the cluster that will need to have docker containers registered to consul and any host that is running as a consul agent. This watches the host on the docker socket and anytime there is a change, registers or deregisters from Consul. 

On each agent:

# docker run -d --name=registrator --net=host --volume=/var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator:latest consul://$IP:$PORT

Make it persistent after reboot with unit file (if using systemd):

# cat /etc/systemd/system/registrator.service
[Unit]
Description=Registrator Container
After=docker.service
Requires=docker.service

[Service]
TimeoutStartSec=0
Restart=on-failure
ExecStart=/usr/bin/docker start registrator

[Install]
WantedBy=multi-user.target


# systemctl enable registrator



5) Now this is where the magic begins. Let's create a json for our Marathon service that will be launched. You are required to the service name defined in the env object. Launching an nginx app with alpine base below:

Name: alpine-nginx.json
{
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "docker-registry:5000/alpine-nginx",
    "network": "BRIDGE",
      "portMappings": [
        { "containerPort": 8050, "hostPort": 0, "servicePort": 8050, "protocol": "tcp" }
      ]
    }
  },
  "id": "alpine-nginx",
  "instances": 1,
  "env":
       { "SERVICE_NAME": "alpine", "SERVICE_TAGS": "alpine" },
  "cpus": 0.5,
  "mem": 100,
  "uris": []

}


6) After you launch the app and it starts on Marathon, check Consul to see if service is registered.



7) Now go back to consul-template server and check out the ha-proxy.cfg file. You service along with its port mappings on Mesos will be there as well.

# cat /etc/haproxy/haproxy.cfg
global
  daemon
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  maxconn 4096

defaults
  log            global
  retries             3
  maxconn          2000
  timeout connect  5000
  timeout client  50000
  timeout server  50000

listen http-in
  bind *:80
  mode tcp
  option tcplog
  balance leastconn

  server mesos-agent01 10.x.x.x:31239 check


Hit the consul-template server at port 80 and you will be routed to your nginx app.

# curl localhost:80
<!DOCTYPE html>
<html>
<body>
<h3>This container is actually running at: </h3>
<p id="demo"> </p>

<script>
var x = location.host;
document.getElementById("demo").innerHTML= x;

</script>

</body>
</html>


8) Scale the app in Marathon to 3 and watch consul-template automatically update your HAProxy config. Yellow is old, green is new.

# cat /etc/haproxy/haproxy.cfg
global
  daemon
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  maxconn 4096

defaults
  log            global
  retries             3
  maxconn          2000
  timeout connect  5000
  timeout client  50000
  timeout server  50000

listen http-in
  bind *:80
  mode tcp
  option tcplog
  balance leastconn
  server mesos-agent01 10.x.x.x:31743 check
  server mesos-agent01 10.x.x.x:31239 check
  server mesos-agent01 10.x.x.x:31577 check

9) Now kill one of the instances from Marathon, this simulates a failure scenario. Consul-template will update the change for the failed instance to the new! Yellow and green are the ones that have existed, blue is the new.

# cat /etc/haproxy/haproxy.cfg
global
  daemon
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  maxconn 4096

defaults
  log            global
  retries             3
  maxconn          2000
  timeout connect  5000
  timeout client  50000
  timeout server  50000

listen http-in
  bind *:80
  mode tcp
  option tcplog
  balance leastconn
  server mesos-agent01 10.x.x.x:31835 check
  server mesos-agent01 10.x.x.x:31239 check

  server mesos-agent01 10.x.x.x:31577 check


Feel free to use your consul-template server for as many other services as you need. All you need to do is add additional service parameters in your template file as before with a different port.

We are calling our consul-template servers "Edge Nodes" as they are actually outside of our Infrastructure and routing to the inside. These can live anywhere on your network as the only thing they need is access to read your Service Discovery. You should be able to dedicate very little resources to these machines as possible 1GB Mem 1 CPU. With the correct setup, you can also run these Edge Nodes in docker containers. You will just need to statically assigned IPs (Flannel, Weave, Calico, etc..) and port mappings for that container. 


Friday, May 6, 2016

Consul Server and Consul Agent Systemd Units

Consul Server and Consul Agent Systemd Units for RHEL/CentOS 7


Consul Server ->/etc/systemd/system/consul-server.service

[Unit]
Description=Consul Server
After=network.target

[Service]
User=root
Group=root
Environment="GOMAXPROCS=2"
ExecStart=/usr/local/bin/consul agent -config-dir /etc/consul.d/server
ExecReload=/bin/kill -9 $MAINPID
KillSignal=SIGINT
Restart=on-failure
RestartSec=1

[Install]

WantedBy=default.target


Consul Agent -> /etc/systemd/system/consul-client.service

[Unit]
Description=Consul Server
After=network.target

[Service]
User=root
Group=root
Environment="GOMAXPROCS=2"
ExecStart=/usr/local/bin/consul agent -config-dir /etc/consul.d/client
ExecReload=/bin/kill -9 $MAINPID
KillSignal=SIGINT
Restart=on-failure


[Install]
WantedBy=multi-user.target

Ultimate Container Sandbox | Isolating Containers in Containers

This was something fun I worked on for while to display how to give users a safe development box to do things like learn, play or test with docker. Its an extremely ephemeral environment and can be rebuilt in secs. It has been sitting in my drafts for a bit but wanted to write about it...... 

Anyone that has been involved in the docker ecosystem over the past several years has more than likely seen the following image below:





Running docker inside of docker. This is nothing new and in fact if you are using Docker universally to run virtually everything such as monitoring or service discovery, chances are you are most likely mounting the docker socket inside your container. I personally use docker in docker to build and push doing the same thing. 

This is where it gets hairy and you get into the inception aspect of this whole mess.

The cool thing with running docker in docker is the fact that you are able to give yourself a nice little test bed with no worries of destroying ready containers and also utilize docker command line at the same time. Building and push new images etc. The only issue with this is the fact that you are mounting the docker socket within the container itself. You are exposing the hosts images and containers to the docker in docker. If you run a '
docker images' inside the docker container, you are seeing the hosts images. If you run a 'docker rm|rmi' you will wipe the host you are running on. There is NO isolation in this. Not only would you wipe the host but anyone else that is running docker in docker on the host would be doing the same thing. 

One way I have figured out how to isolate docker running on the same host is to utilize docker's father project, LXC. By running docker inside of LXC, each LXC instance is completely isolated from the other and you are safely able to utilize docker without affecting anyone else. As with docker, LXC can also be spun up in a matter of seconds so in the event that you do something in LXC that you dont like, blow it away and spin up a new. Good read and another instance of this being used: Openstack Carina Project

Docker on LXC on Linux


Image provide by yours truly... You're Welcome!

Let us get this going. Ubuntu as the underlying host OS as I am starting to go back to my original Linux roots.

1) Install LXC:
apt-get update && apt-get install lxc

2) Create the LXC container and add the following lines to each containers configs /var/lib/lxc/$LXC_NAME/config:
lxc-create -t download -n meh-01 -- -d ubuntu -r trusty -a amd64      
  
Add below lines to /var/lib/lxc/meh-01/config  
lxc.aa_profile = unconfined
lxc.cgroup.devices.allow = a
lxc.cap.drop =


3) Start the LXC container, attach and install the needful to get docker installed in LXC:
# lxc-start -n meh-01 -d 
# lxc-attach -n meh-01

Inside LXC:
# apt-get update && apt-get install wget apparmor docker.io -y

4) Check it out!!! 

FROM LXC:
root@meh-01:~# docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.2.1
Git commit (client): 7c8fca2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.2.1
Git commit (server): 7c8fca2
OS/Arch (server): linux/amd64

root@meh-01:~# docker images
REPOSITORY                       TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
golang                           latest              471e087e791d        2 weeks ago         744 MB


root@meh-01:~# docker run -it golang echo hello world
hello world


FROM HOST:
root@docker-builder:~# docker images
The program 'docker' is currently not installed. You can install it by typing:
apt-get install docker


Docker isn't even installed on the host so the host is not being affected... ***Modify changes to your docker options within LXC if you would like to add things like private registry etc...

Next: Create another LXC container and repeat the above steps and notice you get complete isolation and separate development environments with LXC. Add things into the LXC containers such as ssh and port forwarding on the host so you can SSH to it. 

LXC is the original container runtime that got me interested in containers (my blog from a couple years ago). I will continue to use alongside docker for different things because I think that LXC has some functionality the docker doesn't do as well. For example, running OS containers, LXC is much better. Docker still holds the belt for application containers in my opinion. Be sure to check out Rackspace's CaaS mentioned above. Awesome project and read. I will be following not only what they are doing but Openstack as well. 

CONTAINERIZE ALL THE THINGS