One of the more difficult things to manage as you begin to scale and deploy containers at mass is trying to manage communications and access to your services. Not only is managing communications to your services difficult, but also doing it in a way that makes sense, is static and feels normal for your users. With an orchestration tool such as Mesos, your containers will most likely move from host to host quite often. This is exactly how it should be for environments running large amounts of containers. It shouldn't matter where your container lives and you should also not have to search for it as it moves around. Nor should you have to manage ports as your Infrastructure grows and your apps scale. I believe this to be one of the major pieces to consider when planning your container based environment. Think about the reason that you are considering using containers and then think about how you plan to orchestrate access to them and assume that your services will not be running in the same place tomorrow as they were today. We achieve this solution through a simple mechanism of Service Discovery and Load Balancing.In this post, Ill describe the tools that I have chosen to use in my docker based PaaS solution backed by Apache Mesos. I went with a solution that not only would suit the needs of our Mesos based services but would work along side any docker container that was deployed in the environment. Simply setup a node with load balancing and access to your service discovery and have the users route through this node to access their service.
Demonstrations will be done using Marathon, Consul, consul-template + HaProxy, but as I said there are a ton of projects out there that can be used to help solve this issue.
Components used:
- HAProxy
- consul
- consul-template
- Registrator
Workflow:
- Docker service deployed with Marathon to Mesos
- Registrator running on Mesos Agents registers the service to Consul
- consul-template updates HAProxy with port mappings of service(s) and reloads config
- ACCESS TO SERVICE(S)!!!!
Getting Started. Note: You will need a running Mesos Cluster with Marathon and also a running Consul cluster. See my post for getting a Consul cluster up in 10 minutes here:
1) On a server that you would like to use to proxy traffic, install HAProxy and consul-template
# yum install -y haproxy unzip && cd /usr/local/bin/ && wget -O consul-template.zip wget https://releases.hashicorp.com/consul-template/0.14.0/consul-template_0.14.0_linux_amd64.zip
# unzip consul-template.zip
2) Configure consul-template for HAProxy. It will reload the config each time there is a change with the service such as a scale up, down or a failure.
# mkdir -pv /etc/consul-template/ && cd /etc/consul-template
Create new file /etc/consul-template/consul-haproxy.json which will be the configuration file to manage reloading haproxy anytime there is a change is service discovery.
# cat /etc/consul-template/consul-haproxy.json
consul = "$CONSUL:$PORT"
template {
source = "/etc/haproxy/haproxy.template"
destination = "/etc/haproxy/haproxy.cfg"
command = "systemctl reload haproxy"
}
# cat /etc/haproxy/haproxy.template
global
daemon
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
defaults
log global
retries 3
maxconn 2000
timeout connect 5000
timeout client 50000
timeout server 50000
listen http-in
bind *:80
mode tcp
option tcplog
balance leastconn{{range service "$SERVICE"}}
server {{.Node}} {{.Address}}:{{.Port}} check {{end}}
$SERVICE in the template file above is the service name that you will put as part of a ENV parameter in your Marathon json when you launch. It will register itself in Consul as that service name and anytime there is a change if will reflect the change to haproxy.
# cat /etc/haproxy/haproxy.cfg
global
daemon
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
defaults
log global
retries 3
maxconn 2000
timeout connect 5000
timeout client 50000
timeout server 50000
listen http-in
bind *:80
mode tcp
option tcplog
balance leastconn
3) We can go ahead and start consul-template at this point. Run from command line or from systemd unit to make it permanent.
# consul-template -config /etc/consul-template/consul-haproxy.json
OR
# cat /etc/systemd/system/consul-template.service
[Unit]
Description=Consul Template HA Proxy
After=network.target
[Service]
User=root
Group=root
Environment="GOMAXPROCS=2"
ExecStart=/usr/local/bin/consul-template -config /etc/consul-template/consul-haproxy.json
ExecReload=/bin/kill -9 $MAINPID
KillSignal=SIGINT
Restart=on-failure
[Install]
WantedBy=multi-user.target
# systemctl enable consul-template && systemctl start consul-template
# consul-template -config /etc/consul-template/consul-haproxy.json
OR
# cat /etc/systemd/system/consul-template.service
[Unit]
Description=Consul Template HA Proxy
After=network.target
[Service]
User=root
Group=root
Environment="GOMAXPROCS=2"
ExecStart=/usr/local/bin/consul-template -config /etc/consul-template/consul-haproxy.json
ExecReload=/bin/kill -9 $MAINPID
KillSignal=SIGINT
Restart=on-failure
[Install]
WantedBy=multi-user.target
# systemctl enable consul-template && systemctl start consul-template
4) Registrator must be running on any host in the cluster that will need to have docker containers registered to consul and any host that is running as a consul agent. This watches the host on the docker socket and anytime there is a change, registers or deregisters from Consul.
On each agent:
# docker run -d --name=registrator --net=host --volume=/var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator:latest consul://$IP:$PORT
# docker run -d --name=registrator --net=host --volume=/var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator:latest consul://$IP:$PORT
Make it persistent after reboot with unit file (if using systemd):
# cat /etc/systemd/system/registrator.service
[Unit]
Description=Registrator Container
After=docker.service
Requires=docker.service
[Service]
TimeoutStartSec=0
Restart=on-failure
ExecStart=/usr/bin/docker start registrator
[Install]
WantedBy=multi-user.target
# systemctl enable registrator
# cat /etc/systemd/system/registrator.service
[Unit]
Description=Registrator Container
After=docker.service
Requires=docker.service
[Service]
TimeoutStartSec=0
Restart=on-failure
ExecStart=/usr/bin/docker start registrator
[Install]
WantedBy=multi-user.target
# systemctl enable registrator
5) Now this is where the magic begins. Let's create a json for our Marathon service that will be launched. You are required to the service name defined in the env object. Launching an nginx app with alpine base below:
Name: alpine-nginx.json
{
"container": {
"type": "DOCKER",
"docker": {
"image": "docker-registry:5000/alpine-nginx",
"network": "BRIDGE",
"portMappings": [
{ "containerPort": 8050, "hostPort": 0, "servicePort": 8050, "protocol": "tcp" }
]
}
},
"id": "alpine-nginx",
"instances": 1,
"env":
{ "SERVICE_NAME": "alpine", "SERVICE_TAGS": "alpine" },
"cpus": 0.5,
"mem": 100,
"uris": []
}
6) After you launch the app and it starts on Marathon, check Consul to see if service is registered.
Name: alpine-nginx.json
{
"container": {
"type": "DOCKER",
"docker": {
"image": "docker-registry:5000/alpine-nginx",
"network": "BRIDGE",
"portMappings": [
{ "containerPort": 8050, "hostPort": 0, "servicePort": 8050, "protocol": "tcp" }
]
}
},
"id": "alpine-nginx",
"instances": 1,
"env":
{ "SERVICE_NAME": "alpine", "SERVICE_TAGS": "alpine" },
"cpus": 0.5,
"mem": 100,
"uris": []
}
6) After you launch the app and it starts on Marathon, check Consul to see if service is registered.
7) Now go back to consul-template server and check out the ha-proxy.cfg file. You service along with its port mappings on Mesos will be there as well.
# cat /etc/haproxy/haproxy.cfg
global
daemon
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
defaults
log global
retries 3
maxconn 2000
timeout connect 5000
timeout client 50000
timeout server 50000
listen http-in
bind *:80
mode tcp
option tcplog
balance leastconn
server mesos-agent01 10.x.x.x:31239 check
9) Now kill one of the instances from Marathon, this simulates a failure scenario. Consul-template will update the change for the failed instance to the new! Yellow and green are the ones that have existed, blue is the new.
# cat /etc/haproxy/haproxy.cfg
global
daemon
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
defaults
log global
retries 3
maxconn 2000
timeout connect 5000
timeout client 50000
timeout server 50000
listen http-in
bind *:80
mode tcp
option tcplog
balance leastconn
server mesos-agent01 10.x.x.x:31835 check
server mesos-agent01 10.x.x.x:31239 check
server mesos-agent01 10.x.x.x:31577 check
# cat /etc/haproxy/haproxy.cfg
global
daemon
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
defaults
log global
retries 3
maxconn 2000
timeout connect 5000
timeout client 50000
timeout server 50000
listen http-in
bind *:80
mode tcp
option tcplog
balance leastconn
server mesos-agent01 10.x.x.x:31239 check
Hit the consul-template server at port 80 and you will be routed to your nginx app.
# curl localhost:80
<!DOCTYPE html>
<html>
<body>
<h3>This container is actually running at: </h3>
<p id="demo"> </p>
<script>
var x = location.host;
document.getElementById("demo").innerHTML= x;
</script>
</body>
</html>
8) Scale the app in Marathon to 3 and watch consul-template automatically update your HAProxy config. Yellow is old, green is new.
# cat /etc/haproxy/haproxy.cfg
global
daemon
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
defaults
log global
retries 3
maxconn 2000
timeout connect 5000
timeout client 50000
timeout server 50000
listen http-in
bind *:80
mode tcp
option tcplog
balance leastconn
server mesos-agent01 10.x.x.x:31743 check
server mesos-agent01 10.x.x.x:31239 check
server mesos-agent01 10.x.x.x:31577 check
9) Now kill one of the instances from Marathon, this simulates a failure scenario. Consul-template will update the change for the failed instance to the new! Yellow and green are the ones that have existed, blue is the new.
# cat /etc/haproxy/haproxy.cfg
global
daemon
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
defaults
log global
retries 3
maxconn 2000
timeout connect 5000
timeout client 50000
timeout server 50000
listen http-in
bind *:80
mode tcp
option tcplog
balance leastconn
server mesos-agent01 10.x.x.x:31835 check
server mesos-agent01 10.x.x.x:31239 check
server mesos-agent01 10.x.x.x:31577 check
Feel free to use your consul-template server for as many other services as you need. All you need to do is add additional service parameters in your template file as before with a different port.
We are calling our consul-template servers "Edge Nodes" as they are actually outside of our Infrastructure and routing to the inside. These can live anywhere on your network as the only thing they need is access to read your Service Discovery. You should be able to dedicate very little resources to these machines as possible 1GB Mem 1 CPU. With the correct setup, you can also run these Edge Nodes in docker containers. You will just need to statically assigned IPs (Flannel, Weave, Calico, etc..) and port mappings for that container.
