The Prometheus Operator for Kubernetes provides easy monitoring definitions for Kubernetes services and deployment and management of Prometheus instances.

This have been tested on a hybrid ARM64 / X84-64 Kubernetes cluster deployed as this article.

This repository collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

The content of this project is written in jsonnet and is an extension of the fantastic kube-prometheus project.

To continue using my previous stack with manifests and previous versions of the operator and components, use the legacy repo tag from: https://github.com/carlosedp/prometheus-operator-ARM/tree/legacy.

Components included in this package:

• The Prometheus Operator
• Highly available Prometheus
• Highly available Alertmanager
• Prometheus node-exporter
• kube-state-metrics
• CoreDNS
• Grafana
• SMTP relay to Gmail for Grafana notifications

There are additional modules (enabled by default) to monitor other components of the infra-structure. These can be disabled on vars.jsonnet file by setting the module in installModules to false.

The additional modules are:

• ARM_exporter to generate temperature metrics
• MetalLB metrics
• Traefik metrics
• ElasticSearch metrics
• APC UPS metrics

There are also options to set the ingress domain suffix and enable persistence for Grafana and Prometheus.

The repository already provides a set of compiled manifests to be applied into the cluster. The deployment can be customized thru the jsonnet files.

To simply deploy the stack, run:

$ make deploy

# Or manually:

$ kubectl apply -f manifests/

# It can take a few seconds for the above 'create manifests' command to fully create the following resources, so verify the resources are ready before proceeding.
$ until kubectl get customresourcedefinitions servicemonitors.monitoring.coreos.com ; do date; sleep 1; echo ""; done
$ until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done

$ kubectl apply -f manifests/ # This command sometimes may need to be done twice (to workaround a race condition).

The content of this project consists of a set of jsonnet files making up a library to be consumed.

The project requires json-bundler and the jsonnet compiler. The Makefile does the heavy-lifting of installing them. You need Go already installed:

git clone https://github.com/carlosedp/prometheus-operator-ARM
cd prometheus-operator-ARM
make vendor
# Change the jsonnet files...
make

After this, a new customized set of manifests is built into the manifests dir. To apply to your cluster, run:

make deploy

To uninstall, run:

make teardown

This project depends on the following images (all supports ARM, ARM64 and AMD64 thru manifests):

Alertmanager Blackbox_exporter Node_exporter Snmp_exporter Prometheus

• Source: https://github.com/carlosedp/prometheus-ARM
• Autobuild: https://travis-ci.org/carlosedp/prometheus-ARM
• Images:
  • https://hub.docker.com/r/carlosedp/prometheus/
  • https://hub.docker.com/r/carlosedp/alertmanager/
  • https://hub.docker.com/r/carlosedp/blackbox_exporter/
  • https://hub.docker.com/r/carlosedp/node_exporter/
  • https://hub.docker.com/r/carlosedp/snmp_exporter/

ARM_exporter

• Source: https://github.com/carlosedp/docker-arm_exporter
• Autobuild: https://travis-ci.org/carlosedp/docker-arm_exporter
• Images: https://hub.docker.com/r/carlosedp/arm_exporter/

Prometheus-operator

• Source: https://github.com/carlosedp/prometheus-operator
• Autobuild: No autobuild yet. Use provided build_images.sh script.
• Images: https://hub.docker.com/r/carlosedp/prometheus-operator

Prometheus-adapter

• Source: https://github.com/DirectXMan12/k8s-prometheus-adapter
• Autobuild: No autobuild yet. Use provided build_images.sh script.
• Images: https://hub.docker.com/r/carlosedp/k8s-prometheus-adapter

Grafana

• Source: https://github.com/carlosedp/grafana-ARM
• Autobuild: https://travis-ci.org/carlosedp/grafana-ARM
• Images: https://hub.docker.com/r/grafana/grafana/

Kube-state-metrics

• Source: https://github.com/kubernetes/kube-state-metrics
• Autobuild: No autobuild yet. Use provided build_images.sh script.
• Images: https://hub.docker.com/r/carlosedp/kube-state-metrics

Addon-resizer

• Source: https://github.com/kubernetes/autoscaler/tree/master/addon-resizer
• Autobuild: No autobuild yet. Use provided build_images.sh script.
• Images: https://hub.docker.com/r/carlosedp/addon-resizer

Obs. This image is a clone of AMD64, ARM64 and ARM with a manifest. It's cloned and generated by the build_images.sh script

configmap_reload

• Source: https://github.com/carlosedp/configmap-reload
• Autobuild: https://travis-ci.org/carlosedp/configmap-reload
• Images: https://hub.docker.com/r/carlosedp/configmap-reload

prometheus-config-reloader

• Source: https://github.com/coreos/prometheus-operator/tree/master/contrib/prometheus-config-reloader
• Autobuild: No autobuild yet. Use provided build_images.sh script.
• Images: https://hub.docker.com/r/carlosedp/prometheus-config-reloader

SMTP-server

• Source: https://github.com/carlosedp/docker-smtp
• Autobuild: https://travis-ci.org/carlosedp/docker-smtp
• Images: https://hub.docker.com/r/carlosedp/docker-smtp

Kube-rbac-proxy

• Source: https://github.com/brancz/kube-rbac-proxy
• Autobuild: No autobuild yet. Use provided build_images.sh script.
• Images: https://hub.docker.com/r/carlosedp/kube-rbac-proxy