Monitoring Docker Swarm with Prometheus and ELK stack.

This repository describes and publishes our setup of monitoring a Docker Swarm with the help of the ELK repository and Prometheus with it's scrapers.

As of version 2.x.x. of Prometheus they changed the user to nobody. This means that if you use a persistent data directory is has to be chmod'ed to 777. In this example this will be the directory /var/dockerdata/prometheus/data.

• Ubuntu (16.04 or higher) or RHEL host(s) or CentOS7 hosts
• Docker v1.13.1 (minimum) (Latest full run on 17.12.0-ce)
  • Experimental Mode must be set to true (to be able to use "docker deploy" with compose v3 files)
  • Must run in Swarm Mode
  • 2 overlay networks ("monitoring" and "logging")

We have split up the monitoring into 2 basic parts:

Create the following directories:

• /var/dockerdata/prometheus
• /var/dockerdata/elasticsearch/data
• /var/dockerdata/elastalert/logs
• /var/dockerdata/alertmanager/data
• /var/dockerdata/grafana
• /var/dockerdata/curator

Host setting for ElasticSearch (Look here for more information)

$ sysctl -w vm.max_map_count=262144

Changing mod of persistent data directory (Only for Prometheus 2.x.x)

$ chmod 777 /var/dockerdata/prometheus/data

$ docker swarm init
$ docker network create -d overlay monitoring
$ docker network create -d overlay logging

Make sure to look at the compose files for the volume mappings. In this example everything is mapped to /var/dockerdata/{servicename}/{directories}. Adjust this to your own liking or create the same structure as used in this example.

$ docker deploy --compose-file docker-compose-logging.yml logging

$ docker deploy --compose-file docker-compose-monitoring.yml monitoring

In order to get the logs from the services/containers to Logstash you need to start them with a different logdriver.

Compose file:

logging:
    driver: gelf
    options:
        gelf-address: "udp://127.0.0.1:12201"
        tag: "<name of container for filtering in elasticsearch>" 

Run command:

$ docker run \
         --log-driver=gelf \
         --log-opt gelf-address=udp://127.0.0.1:12201 \
         --log-opt tag="<name of container for filtering in elasticsearch>" \
         ....
         ....

Basilio Vera's repo's (https://hub.docker.com/u/basi/) have been used for information. This got me a long way with building up a monitoring stack. Also using his version of Node-Exporter and some alert files so we have access to HOST_NAME and some startup alerts. He made a really nice Grafana Dashboard too which we used as a base. You can check it out here (https://grafana.net/dashboards/609).

The files are free to use and you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation.