Marathon-lb: Service Discovery and Loadbalancing for containers running on Mesos/Marathon
Marathon is a production-grade Docker orchestrator for Docker containers which runs on top of apache Mesos. It enables users to run Docker containers on Apache’s popular Mesos Cluster
Marathon Features
1. Supports Multiple container runtimes : Marathon has first-class support for both Mesos containers (using cgroups) and Docker.
2. Stateful Apps : Marathon can bind persistent storage volumes to your application. You can run databases like MySQL , and have storage accounted for by Mesos.
3. Healthchecks and Autorecovery : Evaluate your application’s health using HTTP or TCP checks and relaunch faulty instances
4. Service Discovery and registration : Supply an HTTP endpoint to receive notifications – for example to integrate with an external load balancer.
5. Monitoring useful Metrics : Query metrics at /metrics in JSON format or push them to systems like graphite, statsd and Datadog
6. Complete RESTful API : Enables you to integrate it with other applications and scripts
7. Powerful and Beautiful GUI
In this post we will discuss Marathon-lb , Marathon-lb provides a port based service discovery and load balancing for docker containers launched using Marathon framework. It uses haproxy, with lua support and Marathon’s app state endpoint to get things working. We will go through a simple example of a web cluster of nginx webserver, launched using Marathon framework running top of mesos cluster. See the following Marathon application named “mywebservernew”
Above application was launched using following Marathon configuration
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | { “id”: “/mywebservernew”, “cmd”: “nginx -g ‘daemon off;'”, “cpus”: 0.5, “mem”: 64, “disk”: 0, “instances”: 3, “container”: { “type”: “DOCKER”, “volumes”: [], “docker”: { “image”: “nginx:stable”, “network”: “BRIDGE”, “portMappings”: [ { “containerPort”: 80, “hostPort”: 0, “servicePort”: 10004, “protocol”: “tcp”, “labels”: {} } ], “privileged”: false, “parameters”: [], “forcePullImage”: false } }, “healthChecks”: [ { “path”: “/”, “protocol”: “HTTP”, “portIndex”: 0, “gracePeriodSeconds”: 5, “intervalSeconds”: 20, “timeoutSeconds”: 20, “maxConsecutiveFailures”: 3, “ignoreHttp1xx”: false } ], “labels”: { “HAPROXY_GROUP”: “external” }, “portDefinitions”: [ { “port”: 10004, “protocol”: “tcp”, “labels”: {} } ] } |
We have launched 3 instances of nginx using Marathon, from the above image its clear there are 3 container instances running and they are listening on following ports on Mesos slave nodes
| 1 2 3 4 5 | 192.168.2.8:31502 192.168.2.8:31281 192.168.2.16:31677 |
These Docker containers run on HA mode, if any of the mesos slave goes down or any container instance fails, it would be automatically relanched on a healthy slave. So we don’t have to worry about maintaining High availability, marathon takes care of this.
Whats Next?
Yes, we need to strat sending traffic to these Docker container instances, which are lanched using Marathon. Docker containers exposed following ports for the nginx webserver running in it
| 1 2 3 4 5 | 192.168.2.8:31502 192.168.2.8:31281 192.168.2.16:31677 |
How we distribute traffic to it? With highly elastic nature of the Containers and Marathon, port numbers and IPs can change at any time. It can be due to a hardware issue on any of the Mesos slave or a crashed container instance. We need to make sure that our loadbalancer is capable of detecting those changes in minimal time and route the traffic accordingly. Mesosphere has Marathon-lb to address this concern. Marathon-lb is a tool for managing HAProxy by consuming Marathon’s app state. HAProxy is a fast, efficient, battle-tested, highly available load balancer with many advanced features like healthchecks, SSL, session persistance ..
How Marathon-lb works ?
The marathon-lb script marathon_lb.py connects to the marathon API to retrieve all running apps, generates a HAProxy config and reloads HAProxy. Whenever there is any changes in
app state , Marathon-lb updates the haproxy configuration file with latest IP and Port number and reloads haproxy. By default, marathon-lb binds to the service port of every application and sends incoming requests to the application instances.
How to Invoke Marathon-lb?
Marathon-lb relies on latest lua support embedded in haproxy, to generate the config file. You need to download and install Haproxy with Lua support, haproxy version 1.6.0 + above is needed
How to compile Haproxy with Lua support ?
You can follow instructions at this link
Start Marathon-lb
After you installed haproxy with lua enabled, you can download Marathon-lb scripts from git repo
Https://github.com/mesosphere/marathon-lb
| 1 2 3 | git clone https://github.com/mesosphere/marathon-lb.git |
Above command will clone the repo to folder marathon-lb
Type
| 1 2 3 4 | cd marathon–lb ./marathon_lb.py —marathon http://192.168.2.21:8080 –group external –dont-bind-http-https |
By running this command , we are asking Marathon-lb to fetch the app state from Marathon state api http://192.168.2.21:8080 , create haproxy configuration and reload haproxy
Here is the sample configuration file generated for the “mywebservernew” webapp which we discussed earlier
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | frontend mywebservernew_10004 bind *:10004 mode http use_backend mywebservernew_10004 backend mywebservernew_10004 balance roundrobin mode http option forwardfor http–request set–header X–Forwarded–Port %[dst_port] http–request add–header X–Forwarded–Proto https if { ssl_fc } option httpchk GET / timeout check 20s server 192_168_2_16_31677 192.168.2.16:31677 check inter 20s fall 4 server 192_168_2_8_31281 192.168.2.8:31281 check inter 20s fall 4 server 192_168_2_8_31502 192.168.2.8:31502 check inter 20s fall 4 |
As you can see in the haproxy configuration, there is a frontend binding on port 10004 with three backends which are our docker containers. This frontend port number is customizable If we relaunch the application, marathon-lb detect the changes and automatically update the configuration and reload haproxy. The best part is , haproxy updates are almost realtime. This way it handle the failures with minimal downtime. There are many more features available in marathon-lb, advice you to go through the documentation at https://github.com/mesosphere/marathon-lb .