Prometheus alert tool

About The Project

This project was implemented example of an alert tool based of Prometheus and Alert Manager.

Built With

Getting Started

metric-generator is implemented with spring boot. The aim of metric-generator is to expose three endpoints for managing (increase, reduce and reset) a custom metric (micrometer). The custom metric is temporarily saved in a local cache.

Through Spring Boot-Actuator, metric-generator expose an endpoint for that Prometheus do scraping and collect the metrics (custom and technical metrics)

The scraping options are configured into prometheus.yml concretely in first section scrape_configs.

scrape_configs:
  - job_name: 'metric-generator'
    scrape_interval: 2s
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: [ 'metric-generator-one:8080','metric-generator-two:8080' ]

Prometheus is a collector and a time series database. Into monitoring ecosystem of Prometheus, Alert Manager is a tool that it allows creating the rules based in PromQL (Prometheus Query Language, language query for time series). Depends on configuration, when the alert is fired, at the same time, is notified of different ways, for example, sending an email.

For connecting an instance of Prometheus with Alert Manager only is necessary declare into prometheus.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

The rules of alerts are declared into rules.yml and linked with prometheus into prometheus.yml:

rules.yml

groups:
  - name: rules
    rules:
      - alert: RequestMoreThanFive
        expr:  sum(custom_metric) > 5
        for: 1s
        labels:
          severity: high
        annotations:
          summary: Application receiving too many requests in endpoint /inc

prometheus.yml

rule_files:
  - './rules.yml'

To end, the environment config only need declare how alerts will be sent and for whom. This is done into alertmanager.yml

route:
  receiver: emailer
receivers:
  - name: emailer
    email_configs:
      - to: [email protected]
        from: [email protected]
        require_tls: false
        smarthost: service-mail:25

Prerequisites

All project is executed and orchestrated with containers and docker-compose so it's necessary has got install docker and docker-compose.

By default, the alerts are sent by email, hence it is needed two temporal emails, for example:

To facilitate email config into the yml file, only it is necessary executing set-email.sh and follow the instructions.

./set-email.sh

Architecture

Installation

Clone the repo

   git clone https://github.com/ivanas93/prometheus-alert-tool prometheus-alert-tool
   cd prometheus-alert-tool

Compile metric-generator
```
cd metric-generator
mvn clean compile
```
Configure emails sender and receiver
```
cd ..
./set-email.sh
```
Run docker-compose
```
docker-compose up
```

Usage

By default, when custom alert arrives to count equals five, the alert is fired by Prometheus. For this behaviour follows the next instructions

Call the endpoint inc until valueMetric:5

curl -v http://localhost:8080/inc

{"valueMetric":5}

Go to prometheus-ui-alerts and verify that the alert was fired after was evaluated 2.1 Without events 2.2 evaluating alert 2.3 fired alert
Go to alert-manager-ui and verify that alert was send to receivers.

Bonus

Another way of microservice alerting is across Grafana. For this option , It's had added a new service into docker-compose. This service is configured to aggregate into a container a data source configuration and a pre-configured dashboard. After use docker-compose up , only it is necessary open grafana-ui.

ivanas93 / prometheus-alert-tool Goto Github PK