Code Monkey home page Code Monkey logo

helm-monitor's Introduction

Helm Monitor plugin

Monitor a release, rollback to a previous version depending on the result of a PromQL (Prometheus), events (Sentry), Lucene or DSL query (Elasticsearch).

Helm monitor failure

Demo

asciicast

Install

$ helm plugin install https://github.com/ContainerSolutions/helm-monitor

Usage

Helm monitor diagram

A rollback happen only if the number of result from the query is greater than 0.

You can find a step-by-step example in the ./examples directory.

Prometheus

Monitor the peeking-bunny release against a Prometheus server, a rollback is initiated if the 5xx error rate is over 0 as measured over the last 5 minutes.

$ helm monitor prometheus peeking-bunny 'rate(http_requests_total{code=~"^5.*$"}[5m]) > 0'

You can connect to a given Prometheus instance, by default it will connect to http://localhost:9090.

$ helm monitor prometheus --prometheus=http://prometheus:9090 \
    peeking-bunny \
    'rate(http_requests_total{code=~"^5.*$"}[5m]) > 0'

Elasticsearch

Monitor the peeking-bunny release against an Elasticsearch server, a rollback is initiated if the 5xx error rate is over 0 for the last minute.

Using a Lucene query:

$ helm monitor elasticsearch peeking-bunny 'status:500 AND kubernetes.labels.app:app AND version:2.0.0'

Using a query DSL file:

$ helm monitor elasticsearch peeking-bunny ./query.json

You can connect to a given Elasticsearch instance, by default it will connect to http://localhost:9200.

$ helm monitor elasticsearch --elasticsearch=http://elasticsearch:9200 \
    peeking-bunny \
    'status:500 AND kubernetes.labels.app:app AND version:2.0.0'

Sentry

Monitor the peeking-bunny release against a Sentry server, a rollback is initiated if the number of events is over 0 for the release 2.0.0:

$ helm monitor sentry my-app \
    --api-key <SENTRY_API_KEY> \
    --organization sentry \
    --project my-project \
    --sentry http://sentry:9000 \
    --tag release=2.0.0 \
    --regexp
    'Error with database connection.*'

Docker

You can also use the Helm monitor backed Docker image to monitor:

$ docker run -ti -v $HOME/.kube:/root/.kube containersol/helm-monitor \
    monitor prometheus --prometheus=http://prometheus:9090 my-release \
    'rate(http_requests_total{code=~"^5.*$"}[5m]) > 0'

Development

Require Go >= 1.11.

# Clone the repo, then add a symlink to the Helm plugin directory:
$ ln -s $GOPATH/src/github.com/ContainerSolutions/helm-monitor ~/.helm/plugins/helm-monitor

# Build:
$ GOPATH="" GO111MODULE=on go build -o helm-monitor ./cmd/...

# Run:
$ helm monitor elasticsearch my-release ./examples/elasticsearch-query.json

Alternatives

  • Kuberbs - Kubernetes Automatic Rollback System

helm-monitor's People

Contributors

etiennetremel avatar markomalis avatar thbkrkr avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

helm-monitor's Issues

Trying to get in touch regarding a security issue

Hello ๐Ÿ‘‹

I run a security community that finds and fixes vulnerabilities in OSS. A researcher (@arshadkazmi42) has found a potential issue, which I would be eager to share with you.

Could you add a SECURITY.md file with an e-mail address for me to send further details to? GitHub recommends a security policy to ensure issues are responsibly disclosed, and it would help direct researchers in the future.

Looking forward to hearing from you ๐Ÿ‘

(cc @huntr-helper)

Multiple Queries to Prometheus?

Does helm-monitor currently support multiple queries? Meaning, instead of just checking for example http 500, can we also check for latency going up etc? Would that be done just by executing another helm-monitor command?

x509: certificate signed by unknown authority querying external ES server

Hi, I am trying to monitor my helm releases with this awesome plugin, but I am trying to query an ES server that is external outside my k8 cluster, its on the same network though. This is the command I am trying:

helm monitor elasticsearch --elasticsearch=https://$ES_SERVER_URL:9200 "$APP_NAMESPACE" 'kubernetes.namespace:ecommerce-registryv2-uat AND message:"Error creating bean'

And I get this error:

x509: certificate signed by unknown authority

So I am trying to some how ask helm monitor to ignore the cert on that ES, but I am not sure how to do that.

You help will be much appreciated.

'helm monitor' returns `no such file or directory` error

Hi, I am trying to use the plugin but I get the following error.

$ helm plugin install https://github.com/ContainerSolutions/helm-monitor
/usr/bin/wget
Connecting to github.com (192.30.253.112:443)
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (52.216.224.32:443)
helm-monitor_linux_0   8% |**                             |   339k  0:00:11 ETA
helm-monitor_linux_0  38% |***********                    |  1597k  0:00:03 ETA
helm-monitor_linux_0 100% |*******************************|  4179k  0:00:00 ETA
Installed plugin: monitor
$ helm monitor
Error: fork/exec /root/.helm/plugins/helm-monitor/helm-monitor: no such file or directory

I try mkdir -p /root/.helm/plugins/helm-monitor/helm-monitor but I get
mkdir: can't create directory '/root/.helm/plugins/helm-monitor/helm-monitor': File exists

The container I try to do the above is based on alpine:latest .

Any help would be appreciated!

Helm monitor plugin not working "Error: context deadline exceeded"

Hi Team ,

We are facing issue while using helm monitor plugin. Prometheus server is up and running. But when we are using it with helm monitor commands its not working. We have tried with different PROMQL query as well but still same result.

prometheus server :-
curl http://IP:PORT/-/healthy
Prometheus Server is Healthy.

helm monitor prometheus --prometheus=http://IP:PORT my-release 'rate(http_requests_total{code=~"^5.*$"}[5m]) >
0'
Error: context deadline exceeded
Usage:
monitor prometheus [flags] RELEASE PROMQL

Flags:
-h, --help help for prometheus
--prometheus string prometheus address (default "http://localhost:9090")

Global Flags:
--dry-run simulate a rollback if triggered by query result
--expected-result-count int number of results that are expected to be returned by the query (rollback triggered if the number of results exceeds this value)
--force force resource update through delete/recreate if needed
-i, --interval int time in seconds between each query (default 10)
--no-hooks prevent hooks from running during rollback
--rollback-timeout int time in seconds to wait for any individual Kubernetes operation during the rollback (like Jobs for hooks) (default 300)
--timeout int time in seconds to wait before assuming a monitoring action is successfull (default 300)
-v, --verbose enable verbose output
--wait if set, will wait until all Pods, PVCs, Services, and minimum number of Pods of a Deployment are in a ready state before marking a rollback as successful. It will wait for as long as --rollback-timeout

Error: plugin "monitor" exited with error

We have installed Elasticsearch also and tried with that , getting the same output with Elasticsearch also.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.