Wicked Awesome Tech: Multihost Docker Networking

One of the major issues that people have with running docker is the fact that docker and containers natively only support localhost networking capabilities. So this means that by default, only the localhost and its services know about the containers. Host A will not be able to communicate with Host B by default. Here is a quick demo of how to use CoreOS's networking project Flannel, so that you can have a multi-host docker environment where all the hosts and their containers can commuincate.

More info on Flannel: https://github.com/coreos/flannel

More info on ETCD: https://github.com/coreos/etcd

You will need to have 1 or more etcd server(s). We will be using a single node for this demo.

On etcd Server(s)

1. Install etcd

2. Configure etcd - /etc/etcd/etcd.conf

# cat /etc/etcd/etcd.conf | grep -v '^#'

ETCD_NAME=default

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

ETCD_LISTEN_PEER_URLS="http://0.0.0.0:7001,http://localhost:2380"

ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"

ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379"

3. Enable and start etcd

# systemctl enable etcd && systemctl start etcd

4. Define etcd network:

etcdctl mk /blah.com/network/config '{"Network":"172.17.0.0/16"}'

You should be able to get the json for that key.

# curl -s -L http://ETCD_SERVER:2379/v2/keys/blah.com/network/config | python -m json.tool

On worker/slave/client(s)… any machine that is going to have flannel running

1. Install flannel

2. Configure flannel - /etc/sysconfig/flanneld

# cat /etc/sysconfig/flanneld | grep -v '^#'

FLANNEL_ETCD="http://ETCD_SERVER:2379"

FLANNEL_ETCD_KEY="/blah.com/network"

3. If Docker is already installed, stop and remove docker interface.

# systemctl stop docker

ip link delete docker0

systemctl start flanneld && systemctl enable flanneld

systemctl start docker

Rinse and repeat for all other desired docker hosts:

You should be able to see the networking configs and subnets being created.

# curl -s -L http://master:2379/v2/keys/blah.com/network/config | python -m json.tool

# curl -s -L http://master:2379/v2/keys/blah.com/network/subnets | python -m json.tool

You should now be able to ping containers between different hosts!

After starting a few containers on different hosts, try it out. Each host gets its own subnet if you notice below.

Host A:

# docker inspect f41cd57b4ef5 | grep -i ipaddress

"IPAddress": "172.17.24.3",

"SecondaryIPAddresses": null,

Host B:

# docker inspect 1b5b48c6be47 | grep -i ipaddress

"IPAddress": "172.17.80.16",

"SecondaryIPAddresses": null,

From Host A container, ping container 1b5b48c6be47 on Host B:

# docker exec -it f41cd57b4ef5 ping 172.17.80.16

PING 172.17.80.16 (172.17.80.16): 56 data bytes

64 bytes from 172.17.80.16: seq=0 ttl=62 time=2.336 ms

64 bytes from 172.17.80.16: seq=1 ttl=62 time=0.438 ms

64 bytes from 172.17.80.16: seq=2 ttl=62 time=0.506 ms

Host A is even able to ping container on Host B:

# ping 172.17.80.16

PING 172.17.80.16 (172.17.80.16) 56(84) bytes of data.

64 bytes from 172.17.80.16: icmp_seq=1 ttl=63 time=0.386 ms

64 bytes from 172.17.80.16: icmp_seq=2 ttl=63 time=0.438 ms

Host B is running a nginx container at port 80. Lets curl port 80 on that container from a container on Host A. (will only work if curl installed on container):

# docker exec -it f41cd57b4ef5 curl 172.17.80.16:80

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0<!DOCTYPE html>

<html>

<head>

<title>Welcome to nginx!</title>

<style>

body {

width: 35em;

margin: 0 auto;

font-family: Tahoma, Verdana, Arial, sans-serif;

}

</style>

</head>

<body>

<h1>Welcome to nginx!</h1>

<p>If you see this page, the nginx web server is successfully installed and

working. Further configuration is required.</p>

<p>For online documentation and support please refer to

<a href="http://nginx.org/">nginx.org</a>.<br/>

Commercial support is available at

<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>

</body>

</html>

100 612 100 612 0 0 321k 0 --:--:-- --:--:-- --:--:-- 597k

And from Host A:

# curl 172.17.80.16:80

<!DOCTYPE html>