gluster / gluster-mixins Goto Github PK

View Code? Open in Web Editor NEW

23.0 9.0 8.0 60 KB

A set of Grafana dashboards and Prometheus alerts for Gluster.

License: GNU Lesser General Public License v2.1

Makefile 16.76% Shell 44.39% Dockerfile 38.84%

gluster prometheus-mixins prometheus-alert gluster-grafana-dashboard kubernetes-mixins jsonnet

gluster-mixins's Introduction

Prometheus Monitoring Mixin for Gluster

A set of Grafana dashboards and Prometheus alerts for Gluster.

The scope of this project is to provide Gluster specific Grafana dashboard configs and Prometheus rule files using Prometheus Mixins.

Prerequisites

Jsonnet [Install Jsonnet]

Jsonnet is a data templating language for app and tool developers.

The mixin project uses Jsonnet to provide reusable and configurable configs for Grafana Dashboards and Prometheus Alerts.
Jsonnet-bundler [Install Jsonnet-bundler]

Jsonnet-bundler is a package manager for jsonnet.
Promtool
1. Download Go (>=1.11) and install it on your system.
2. Setup the GOPATH environment.
3. Run $ go get -d github.com/prometheus/prometheus/cmd/promtool

How to use?

Manually generate configs and rules

You can clone this repository and manually generate Grafana Dashboard configs and Prometheus Rules files, and apply it according to your setup.

$ git clone https://github.com/gluster/gluster-mixins.git
$ cd gluster-mixins

To get dependencies

$ jb install

To generate Prometheus Alert file

$ make prometheus_alerts.yaml

To generate Prometheus Rule file

$ make prometheus_rules.yaml

To generate Grafana Dashboard configs

$ make dashboards_out

The prometheus_alerts.yaml and prometheus_rules.yaml files then needs to be passed to your Prometheus Server, and the files in dashboards_out needs to be passed to your Grafana server.

Background

Prometheus Monitoring Mixin design doc

gluster-mixins's People

Contributors

Stargazers

Watchers

Forkers

umangachapagain anmolsachan shtripat cloudbehl johnstrunk evq rajeshm123 hombre9

gluster-mixins's Issues

travis is failing as jsonnetfmt is now it's own binary

Alerts - Volume, Brick and Thinpool Utilization

Required alerts for:
Capacity utilization - Volume
Capacity utilization - Brick
Capacity utilization - Thinpool

Dashboards - K8s-storage-resources-persistent-volume-glusterfs

OCS (glusterfs) specific dashboard for a Gluster volume

Dependencies required for dashboards and addition of JB to prerequisites

Grafana object builder

Grafana builder is required to create a k8s object. The object is of config type and can be applied directly on the k8s cluster.

The file that will be created out of it will have a k8s header with the grafana JSON. After applying you can see the grafana dashboards.

Volume level rules and metrics should also be dependent on job label along with volume label.

Currently glusterExporterSelector: 'job="gluster-exporter"' is hard coded in configs. This represents a cluster level differentiation.
This should not be present here; dashboards and clusters should consider a cluster level entity (according to metrics JOB label can be considered at that level ). Currently there is no concept of cluster_id in metrics and job is the label that differentiates them.

Dashboards - K8s-storage-resources-provisioner-glusterfs - About

Alerts - OCS Cluster and Cluster Nodes health

Need following status alerts:

Node status (Up/Down)
Container status (Up/Down)
Gluster peer in cluster status (Connected/Disconnected)
Glusterd2 service status (Up/Down)
Cluster status

Grafana - Rebalance and Geo-Replication panel

Recording rule for gluster_volume_utilization

Recording rule for getting gluster volume utilization based on subvolume utilization is required.

Part of /issues/12

Unit Testing for Volume & Brick Utilization Alert

We need to add Unit Tests using Promtool features for Gluster Utilization related Alerts.

Remove namespace creation from example.jsonnet file and also manifests.

Dashboards - K8s-storage-resources-persistent-volume-glusterfs-lvm

Dashboard - K8s / Storage Resources / Provisioner / glusterfs

Expose recording rules for gluster specific metrics

Added heal count split-brain metric to gluster dashboard

Alert for absence of the `gluster-exporter` job itself.

We need to have an alert that notifies about the absence of the exporter job itself in case monitoring is enabled.

Adding Dockerfile to build mixins container

Adding Dockerfile to build mixins container and apply the generated file to the k8s cluster

docker build fails due to missing jb install

jsonnet -S lib/rules.jsonnet > prometheus_rules.yaml
jsonnet -J vendor -m dashboards_out lib/dashboards.jsonnet
RUNTIME ERROR: couldn't open import "grafana-builder/grafana.libsonnet": no match locally or in the Jsonnet library paths.
        lib/../dashboards/k8s-storage-resources-glusterfs-pv.libsonnet:1:11-53  thunk <g>
        lib/../dashboards/k8s-storage-resources-glusterfs-pv.libsonnet:60:7     object <grafanaDashboards>
        lib/../dashboards/defaults.libsonnet:7:17-44    object <dashboards>
        lib/dashboards.jsonnet:4:11-27  object <anonymous>
        During manifestation
make: *** [Makefile:17: dashboards_out] Error 1