Friday, September 30, 2016

Mesos-DNS as your upstream DNS or alonside your Enterprise DNS

In this post I will show you how you can get going with your own external Mesos-DNS that you can either A) use as an upstream DNS server or B) incorporate/forward Mesos Tasks to your upstream DNS server. The reason for using either one of these methods is so that you are able to have machines outside of the Mesos cluster discover DNS of Mesos tasks running internal the Mesos cluster and vice versa. Using either one of these methods will provide a way for all DNS entries on your network to query one another. We are currently using Method B in our enterprise so that our Mesos tasks are able to communicate with other services running outside of the cluster such as our Gitlab server.

We will be using docker to run our Mesos-DNS in both methods running on our Bootstrap server.

NOTE: This is currently being used with version DCOS Open & Enterprise 1.7.x and Mesos-DNS version 0.5.2. It has not been tested or used with the latest releases of DCOS or Mesos-DNS. Read the release notes of latest DCOS in terms of VIPs to DNS. Will be testing this functionality in the near future. This setup also means that you have exposed your private agents to routing from outside the Mesos Network and not just through the use of the public agent which we are hoping to change in the future as well.

I would like to send a shout out to Mesosphere for continuing to make an incredible product and opening up DCOS to the community. What a powerful and fun community to work with! I have been fortunate enough to have been involved in Apache Mesos for the past year and a half and have watched this project grow rapidly. Mesosphere is doing some amazing things that are changing the way that we treat Data Centers and Development. Looking forward to continuing the journey with them!


Method A: Using Mesos-DNS as an Upstream DNS Server for ALL your DNS

In this method you are able to use your Mesos-DNS as a DNS server for all DNS on your network. You can plug in the IP of your Mesos-DNS server in your resolv.conf file or you can run a dig against it. This will give you the IP address of the Mesos Agent where the service is running. You can also get the port address by obtaining the SRV records. 

1) Create and edit the json.config for Mesos-DNS. See parameters for explanation: "resolvers" is very important here. Also, you can make the domain what you please. Default is "mesos".

2) Run it in docker:

3) You can now use this Mesos-DNS as your DNS server. Place the IP of Mesos-DNS in your resolv.conf or dig against its IP.

From the example config.json, your services will run under <service> some examples would be:
"" for the leader, 
"" for a list of your mesos master nodes,
"" for a list of your mesos agents,
"" for marathon and
"" for a service named "nginx" running on marathon root.

Be sure to check out Mesos DNS documentation on the naming

You will also be able to query all DNS from all DNS servers defined in "resolvers". This is what provides you the ability to query both internal Mesos and external DNS.

Method B: Incorporating Mesos-DNS with your Enterprise upstream DNS Server

This method provides you with the same capabilities only this method uses sub zones or sub domains on your upstream DNS server. Best benefit from this method is that this doesn't require any changes to your DNS configuration on your servers. Nobody has to know there is an external Mesos-DNS server out there that is forward Mesos Tasks DNS to it.

I haven't personally setup a sub domain on a DNS server before, but there are several good references out there on how to do it for your specific DNS. From the example, you would create "" as the sub domain on your DNS server.

1) Create the sub domain on your specific DNS provider. This is the only additional step needed from Method A.

2) Follow steps 1-3 above from Method A.


Hopefully this can relieve some of the pain for Service Discovery and DNS if you run any form of Mesos. This is one of the most difficult problems I have had to deal with when it comes to running external services alongside internal Mesos Tasks. Mostly because it requires making newbies happy but remember there are still several ways to accomplish this. In previous project I used a combination of Mesos, Consul and Consul DNS to accomplish the same thing.

The only downside of the solution above is that this is not the preferred method when it comes to DCOS Architecture since all routing from the outside goes through an LB on the public agent. Hoping that eventually we can get away from this with a mix of Minute Man LB in the future and DNS for VIPS and or IP per task. Personally, I am not again routing traffic directly to private agents because of speed and HA but it all depends on your preferred method. Always need to conside HA and port clashing etc when running in an environment like Mesos.

1 comment: