Friday, May 13, 2016

Orchestrating Communications to Docker as you Scale Like a Boss

One of the more difficult things to manage as you begin to scale and deploy containers at mass is trying to manage communications and access to your services. Not only is managing communications to your services difficult, but also doing it in a way that makes sense, is static and feels normal for your users. With an orchestration tool such as Mesos, your containers will most likely move from host to host quite often. This is exactly how it should be for environments running large amounts of containers. It shouldn't matter where your container lives and you should also not have to search for it as it moves around. Nor should you have to manage ports as your Infrastructure grows and your apps scale. I believe this to be one of the major pieces to consider when planning your container based environment. Think about the reason that you are considering using containers and then think about how you plan to orchestrate access to them and assume that your services will not be running in the same place tomorrow as they were today. We achieve this solution through a simple mechanism of Service Discovery and Load Balancing.

In this post, Ill describe the tools that I have chosen to use in my docker based PaaS solution backed by Apache Mesos. I went with a solution that not only would suit the needs of our Mesos based services but would work along side any docker container that was deployed in the environment. Simply setup a node with load balancing and access to your service discovery and have the users route through this node to access their service.

Demonstrations will be done using Marathon, Consul, consul-template + HaProxy, but as I said there are a ton of projects out there that can be used to help solve this issue. 

Components used:


Workflow:
  1. Docker service deployed with Marathon to Mesos
  2. Registrator running on Mesos Agents registers the service to Consul
  3. consul-template updates HAProxy with port mappings of service(s) and reloads config
  4. ACCESS TO SERVICE(S)!!!!



































Getting Started. Note: You will need a running Mesos Cluster with Marathon and also a running Consul cluster. See my post for getting a Consul cluster up in 10 minutes here:

1) On a server that you would like to use to proxy traffic, install HAProxy and consul-template

# yum install -y haproxy unzip && cd /usr/local/bin/ && wget -O consul-template.zip wget https://releases.hashicorp.com/consul-template/0.14.0/consul-template_0.14.0_linux_amd64.zip
   
# unzip consul-template.zip 

2) Configure consul-template for HAProxy. It will reload the config each time there is a change with the service such as a scale up, down or a failure. 

# mkdir -pv /etc/consul-template/ && cd /etc/consul-template

Create new file /etc/consul-template/consul-haproxy.json which will be the configuration file to manage reloading haproxy anytime there is a change is service discovery. 

# cat /etc/consul-template/consul-haproxy.json
consul = "$CONSUL:$PORT"

template {
  source = "/etc/haproxy/haproxy.template"
  destination = "/etc/haproxy/haproxy.cfg"
  command = "systemctl reload haproxy"


}

Create the source and destination files for haproxy based on the config above.

# cat /etc/haproxy/haproxy.template
global
  daemon
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  maxconn 4096

defaults
  log            global
  retries             3
  maxconn          2000
  timeout connect  5000
  timeout client  50000
  timeout server  50000

listen http-in
  bind *:80
  mode tcp
  option tcplog
  balance leastconn{{range service "$SERVICE"}}
  server {{.Node}} {{.Address}}:{{.Port}} check {{end}}

$SERVICE in the template file above is the service name that you will put as part of a ENV parameter in your Marathon json when you launch. It will register itself in Consul as that service name and anytime there is a change if will reflect the change to haproxy.

# cat /etc/haproxy/haproxy.cfg
global
  daemon
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  maxconn 4096

defaults
  log            global
  retries             3
  maxconn          2000
  timeout connect  5000
  timeout client  50000
  timeout server  50000

listen http-in
  bind *:80
  mode tcp
  option tcplog
  balance leastconn 


3) We can go ahead and start consul-template at this point. Run from command line or from systemd unit to make it permanent.

# consul-template -config /etc/consul-template/consul-haproxy.json

OR

# cat /etc/systemd/system/consul-template.service
[Unit]
Description=Consul Template HA Proxy
After=network.target

[Service]
User=root
Group=root
Environment="GOMAXPROCS=2"
ExecStart=/usr/local/bin/consul-template -config /etc/consul-template/consul-haproxy.json
ExecReload=/bin/kill -9 $MAINPID
KillSignal=SIGINT
Restart=on-failure

[Install]
WantedBy=multi-user.target


# systemctl enable consul-template && systemctl start consul-template


4) Registrator must be running on any host in the cluster that will need to have docker containers registered to consul and any host that is running as a consul agent. This watches the host on the docker socket and anytime there is a change, registers or deregisters from Consul. 

On each agent:

# docker run -d --name=registrator --net=host --volume=/var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator:latest consul://$IP:$PORT

Make it persistent after reboot with unit file (if using systemd):

# cat /etc/systemd/system/registrator.service
[Unit]
Description=Registrator Container
After=docker.service
Requires=docker.service

[Service]
TimeoutStartSec=0
Restart=on-failure
ExecStart=/usr/bin/docker start registrator

[Install]
WantedBy=multi-user.target


# systemctl enable registrator



5) Now this is where the magic begins. Let's create a json for our Marathon service that will be launched. You are required to the service name defined in the env object. Launching an nginx app with alpine base below:

Name: alpine-nginx.json
{
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "docker-registry:5000/alpine-nginx",
    "network": "BRIDGE",
      "portMappings": [
        { "containerPort": 8050, "hostPort": 0, "servicePort": 8050, "protocol": "tcp" }
      ]
    }
  },
  "id": "alpine-nginx",
  "instances": 1,
  "env":
       { "SERVICE_NAME": "alpine", "SERVICE_TAGS": "alpine" },
  "cpus": 0.5,
  "mem": 100,
  "uris": []

}


6) After you launch the app and it starts on Marathon, check Consul to see if service is registered.



7) Now go back to consul-template server and check out the ha-proxy.cfg file. You service along with its port mappings on Mesos will be there as well.

# cat /etc/haproxy/haproxy.cfg
global
  daemon
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  maxconn 4096

defaults
  log            global
  retries             3
  maxconn          2000
  timeout connect  5000
  timeout client  50000
  timeout server  50000

listen http-in
  bind *:80
  mode tcp
  option tcplog
  balance leastconn

  server mesos-agent01 10.x.x.x:31239 check


Hit the consul-template server at port 80 and you will be routed to your nginx app.

# curl localhost:80
<!DOCTYPE html>
<html>
<body>
<h3>This container is actually running at: </h3>
<p id="demo"> </p>

<script>
var x = location.host;
document.getElementById("demo").innerHTML= x;

</script>

</body>
</html>


8) Scale the app in Marathon to 3 and watch consul-template automatically update your HAProxy config. Yellow is old, green is new.

# cat /etc/haproxy/haproxy.cfg
global
  daemon
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  maxconn 4096

defaults
  log            global
  retries             3
  maxconn          2000
  timeout connect  5000
  timeout client  50000
  timeout server  50000

listen http-in
  bind *:80
  mode tcp
  option tcplog
  balance leastconn
  server mesos-agent01 10.x.x.x:31743 check
  server mesos-agent01 10.x.x.x:31239 check
  server mesos-agent01 10.x.x.x:31577 check

9) Now kill one of the instances from Marathon, this simulates a failure scenario. Consul-template will update the change for the failed instance to the new! Yellow and green are the ones that have existed, blue is the new.

# cat /etc/haproxy/haproxy.cfg
global
  daemon
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  maxconn 4096

defaults
  log            global
  retries             3
  maxconn          2000
  timeout connect  5000
  timeout client  50000
  timeout server  50000

listen http-in
  bind *:80
  mode tcp
  option tcplog
  balance leastconn
  server mesos-agent01 10.x.x.x:31835 check
  server mesos-agent01 10.x.x.x:31239 check

  server mesos-agent01 10.x.x.x:31577 check


Feel free to use your consul-template server for as many other services as you need. All you need to do is add additional service parameters in your template file as before with a different port.

We are calling our consul-template servers "Edge Nodes" as they are actually outside of our Infrastructure and routing to the inside. These can live anywhere on your network as the only thing they need is access to read your Service Discovery. You should be able to dedicate very little resources to these machines as possible 1GB Mem 1 CPU. With the correct setup, you can also run these Edge Nodes in docker containers. You will just need to statically assigned IPs (Flannel, Weave, Calico, etc..) and port mappings for that container. 


4 comments: