Prometheus is an open-source software application used for event monitoring and alerting. It records real-time metrics in a time series database (allowing for high dimensionality) built using a HTTP pull model, with flexible queries and real-time alerting. The project is written in Go and licensed under the Apache 2 License, with source code available on GitHub, and is a graduated project of the Cloud Native Computing Foundation, along with Kubernetes and Envoy.
Create new virtual machine with Vagrant
vagrant up
vagrant ssh
cd /vagrant_data/
Increase vm.max_map_count value for elasticsearch
sudo sysctl -w vm.max_map_count=262144
docker-compose -f docker-stack-monitoring.yml \
-f docker-stack-exporters.yml -f docker-stack-db.yml \
-f docker-stack-common.yml -f docker-compose.yml up -d
Build docker images
docker build --build-arg BRANCH="develop" -f prometheus/Dockerfile -t "twistedfantasy-prometheus" .
docker build -f alertmanager/Dockerfile -t "twistedfantasy-alertmanager" .
docker build -f grafana/Dockerfile -t "twistedfantasy-grafana" .
docker build -f nginx/Dockerfile -t "twistedfantasy-nginx" .
Activate docker swarm
docker swarm init --advertise-addr 192.168.56.101
docker stack deploy --with-registry-auth -c docker-stack-monitoring.yml \
-c docker-stack-exporters.yml -c docker-stack-db.yml \
-c docker-stack-common.yml twistedfantasy-monitoring
List of stacks
docker stack ls
List of available services
docker service ls
Get logs for a service
docker service logs -f twistedfantasy-monitoring_elasticsearch
Open Grafana in browser
http://192.168.56.101:3000/
Open Prometheus in browser
http://192.168.56.101:9090/
Delete a stack
docker stack rm twistedfantasy-monitoring
Remove
- all stopped containers
- all networks not used by at least one container
- all images without at least one container associated to them
- all build cache
docker system prune --all
Delete elasticsearch volume
docker volume rm twistedfantasy-monitoring_grafana-data twistedfantasy-monitoring_prometheus-data
Deactivate docker swarm
docker swarm leave --force
Delete vagrant virtual machine
vagrant halt
vagrant destroy
Grafana provision mechanism doesn't work very stable. For example, we added 2 dashboards and deployed them. Then we want to update 1 of already existing dashboards or add a new one, in this case grafana will not make automatic provision during next grafana container start. To force this you need to delete this grafana service/container and volume.
https://awesome-prometheus-alerts.grep.to/ --- very good resource with huge amount of configuration examples
https://prometheus.io/docs/instrumenting/exporters/ --- custom official and non official exporters
CAdvisor Exporter
Node Exporter
Nginx Exporter
Postgres Exporter
ElasticSearch Exporter
Redis Exporter
RabbitMQ Exporter
JVM Micrometer Exporter
Provisioning
Prometheus Datasource
CAdvisor Dashboard
Node Dashboard
Postgres Dashboard
ElasticSearch Dashboard
Redis Dashboard
RabbitMQ Dashboard
JVM Micrometer Dashboard
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent