benjojo / alertmanager-discord Goto Github PK

Would you be willing to set up automated docker builds/pushes to docker hub?

I have my own local copy of the image, but it'd be nice to skip the building step and get it straight from you.

Thanks for your hard work! Getting alerts in discord is amazing! Really reduces the cost and complexity of getting alerts on mobile :)

Hello,

i run the alertmanager container with the alertmanager-discord but I have error in the alertmanager container

level=warn ts=2021-01-18T21:35:47.762Z caller=notify.go:674 component=dispatcher receiver=discord_webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"http://localhost:9094\": EOF" level=warn ts=2021-01-18T21:35:47.763Z caller=notify.go:674 component=dispatcher receiver=discord_webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"http://localhost:9094\": EOF"

Did discord have some update that broke formatting?

I set this up and i have a bunch of alerts that only show summary, e.g.

=== Network errors over x ===

=== Disk is full ===

For others, I can see the notification in my desktop (something something error on server x)

but the message shows empty

How do you install this stuff?

Hi,

i want to change the ListenAddress but seems that it doesn't work. Discord Notification is working but not changing the ListenAddress.

Here is my compose file

alertmanager_discord:
image: benjojo/alertmanager-discord
container_name: alertmanager_discord
restart: always
networks:
- monitoring
env_file:
- 'env.discord'
labels:
org.label-schmea.group: "monitoring"

Enviroment File:

LISTEN_ADDRESS=0.0.0.0:9099
DISCORD_WEBHOOK=myurl

Logs

Attaching to alertmanager_discord
alertmanager_discord | 2020-11-30T18:30:47.178861110Z 2020/11/30 18:30:47 Listening on: 0.0.0.0:9094

I found this problem while setting up alertmanager-discord in my cluster and tested that It doesn't send to discord and silent leading to investigation consider following Aleart rules

groups:
- name: customized.container.rules
  rules:
  - alert: containerRestarted
    annotations:
      runbook_url: ''
      summary: "More than 2 restarts in a pod {{ $labels.pod }}"
    expr: delta(kube_pod_container_status_restarts_total[5m]) > 2
    for: 5m
    labels:
      severity: critical

It does trigger discord webhook with annotation description not empty though

groups:
- name: customized.container.rules
  rules:
  - alert: containerRestarted
    annotations:
      runbook_url: ''
      summary: "More than 2 restarts in a pod {{ $labels.pod }}"
      description: "{{ $labels.container }} restarted (current value: {{ $value }}s) times in pod {{ $labels.namespace }}/{{ $labels.pod }}"
    expr: delta(kube_pod_container_status_restarts_total[5m]) > 2
    for: 5m
    labels:
      severity: critical

It doesn't have description then when the rules triggered alertmanager firing to webhooks but It does nothing

consider this example alert output from alertmanager to alertmanager-discord. This example doesn't send to discord because It lack annotation description but It does send after alert rules to send description though

{
  "receiver": "webhook",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "Test",
        "dc": "eu-west-1",
        "instance": "localhost:9090",
        "job": "prometheus24"
      },
      "annotations": {
        "summary": "hello",
+       "description": "test"
      },
      "startsAt": "2018-08-03T09:52:26.739266876+02:00",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://simon-laptop:9090/graph?g0.expr=go_memstats_alloc_bytes+%3E+0\u0026g0.tab=1"                                                                                  
    }
  ],
  "groupLabels": {
    "alertname": "Test",
    "job": "prometheus24"
  },
  "commonLabels": {
    "alertname": "Test",
    "dc": "eu-west-1",
    "instance": "localhost:9090",
    "job": "prometheus24"
  },
  "commonAnnotations": {
    "summary": "hello",
    "description": "some description"
  },
  "externalURL": "http://simon-laptop:9093",
  "version": "4",
  "groupKey": "{}:{alertname=\"Test\", job=\"prometheus24\"}"
}

My question is how should we fix this? Should we just thrown an error when no description or send empty string to discord instead?

Or should we just Update README.md to have a required annotation for alertmanager-discord?

I recently created a webhook URL and noticed the length of the id differed from the expected 18 in main.go:87. We could change the regex to https://discord(?:app)?.com/api/webhooks/[0-9]{18,19}/[a-zA-Z0-9_-]+. If this looks good, im happy to open a tiny PR to resolve this.

The bot is running:

./alertmanager-discord -listen.address 127.0.0.1:9095
2021/04/29 00:14:04 Listening on: 127.0.0.1:9095

alertmanager config:

• match:
  alertname: test
  receiver: 'discord_webhook'

• name: 'discord_webhook'
  webhook_configs:

  • url: 'http://localhost:9095'

also set the variable with "export DISCORD_WEBHOOK=....."

I see that the bot tries to send $something but its not going to the discord chat.

2021/04/29 00:18:19 localhost:9095 - [POST]

Could you help me ?

Hello,

if we want to send alerts to various Discord Channels, each Channel having it's own WebHook, we have to run multiple instances of alertmanager-discord.

It could be nice to have a YAML configuration file storing endpoints and WebHooks.

For instance:

webhooks:
  - name: "channel1"
    webhook: "https://discord.com/api/webhooks/..."
  - name: "channel2"
    webhook: "https://discord.com/api/webhooks/..."

With alertmanager-discord having a routed endpoint on http://127.0.01:9094/channel1 and http://127.0.01:9094/channel2.

This would allow to send alerts to an infinite amount of channels with only one running daemon.

My Go is not top-nocht but I'm willing to try for a PR if there's any chance for it to be accepted.

At the moment, if discord-alertmanager is asked to listen on a port that is already in use, it will exit with status code 0. This means it will not be marked as 'failed' by systemd.

Since it's never meant to exit, you should simply add os.Exit(1) to the end of main.

I want to send some alerts from external exporter via kubernetes prometheus stack to discord. I see the alerts in the alertmanager but they don't appear in discord channel and there are a lot of similar errors in alertmanager-discord logs.

2021/08/12 19:46:39 10.200.129.205:9094 - [GET] 
2021/08/12 19:46:39 Failed to unpack inbound alert request - 

hello i'm trying to use your discord bridge with a helm kube-prometheus-stack, without success:

here is my alertmanager config, the alertmanager-discord-bridge ip address is http://10.100.1.6:9094/

alertmanager:
fullnameOverride: alertmanager
enabled: true
ingress:
enabled: false
config:
global:
resolve_timeout: 5m
route:
group_by: ["alertname"]
group_wait: 20s
group_interval: 5m
repeat_interval: 3h
receiver: "discord_webhook"
receivers:

• name: "discord_webhook"
  webhook_configs:
• url: "10.100.1.6:9094"

Hello,

We have lifted the moratorium on integrations in alertmanager.

Would you be willing to help us bring discord as an official native integration?

Thanks for your answer.

I need some way to test if this is relaying the message to discord. Would the following prometheus call work?

{
"version": "4",
"groupKey": "keygroup",
"status": "firing",
"receiver": "receiver1",
"groupLabels": null,
"commonLabels": null,
"commonAnnotations": null,
"externalURL": "am",
"alerts": [
{
"status": "firing",
"labels": null,
"annotations": null,
"startsAt": "",
"endsAt": "",
"generatorURL": "url"
},
]
}

Right now it returns an empty response body, so i have no idea if this is working.

First of all, thank you for this tool, it does exactly what I need!

Unfortunately, I’m not sure if I am allowed to use it in the context where I need it, I couldn’t find a license anywhere in the repository, not even a reference in the readme or a comment header in the code. From choosealicense.com:

If you find software that doesn’t have a license, that generally means you have no permission from the creators of the software to use, modify, or share the software. Although a code host such as GitHub may allow you to view and fork the code, this does not imply that you are permitted to use, modify, or share the software for any purpose.

See https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/licensing-a-repository for more details.

Hello!

When I receive an alert, there is emoji in the beginning and the end of the title.

The question is: where does this emoji come from in the code? E.g. if I wanted to use another emoji where I would make changes in the code?

alermanager.xml

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'cluster', 'job', 'env', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'discord_webhook'
  routes:
  - match:
      severity: warning
    receiver: 'discord_webhook'
  - match:
      severity: critical
    receiver: 'discord_webhook'
receivers:
- name: 'null'
- name: 'discord_webhook'
  webhook_configs:
  - url: 'http://0.0.0.0:9094'

if i run alertmanager first:
root@watcher:~# systemctl start alertmanager

and then docker image:

root@watcher:~# docker run -p 9094:9094 --env-file envo d6 &

i have this error:

root@watcher:~# docker: Error response from daemon: driver failed programming external connectivity on endpoint nifty_mahavira (fb4aef42f88c4340c205e6fc693cb848d9f715d40f4e624988bcddd1bda8dfb3): Error starting userland proxy: listen tcp4 0.0.0.0:9094: bind: address already in use.
ERRO[0000] error waiting for container: context canceled

[1]+  Exit 125                docker run -p 9094:9094 --env-file envo d6

if first docker, and then alertmanager, i have same trouble with 9094 port:

Jan 15 15:18:09 watcher alertmanager[1463]: level=info ts=2021-01-15T12:18:09.457Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.17.0.1 port=9094
Jan 15 15:18:09 watcher alertmanager[1463]: level=error ts=2021-01-15T12:18:09.458Z caller=main.go:241 msg="unable to initialize gossip mesh" err="create memberlist: Could not set up network transport: failed to obtain an address: Failed to start TCP listener on \"0.0.0.0\" port 9094: listen tcp 0.0.0.0:9094: bind:
Jan 15 15:18:09 watcher systemd[1]: alertmanager.service: Main process exited, code=exited, status=1/FAILURE
Jan 15 15:18:09 watcher systemd[1]: alertmanager.service: Failed with result 'exit-code'.

how i can solve this problem ??

Hello,
I have Alertmanager, Prometheus and the discord-webhook, I think pinned configs are right, but zero alerts in discord. When alert in a fired state alertmanager-discord took POST request in logs. I have no idea how to fix.

This is my alertmanager config:

    global:
    templates:
    - '/etc/alertmanager/*.tmpl'
    route:
      receiver: discord
      group_by: ['alertname']
      group_wait: 10s
      repeat_interval: 30m
 
    receivers:
    - name: discord
      webhook_configs:
      - url: 'http://alertmanager-discord.newmonitoring:9094'
        send_resolved: true

This is Alertmanager-discord deployment:

omitted..
      - name: alertmanager-discord
        image: benjojo/alertmanager-discord
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 9094
        env:
          - name: DISCORD_WEBHOOK
            value: 'https://discord.com/api/webhooks/...'
---
apiVersion: v1
kind: Service
metadata:
  name: alertmanager-discord
  namespace: newmonitoring
spec:
  selector:
    app: alertmanager-discord
  ports:
  - port: 9094
    targetPort: 9094
  type: ClusterIP

And this prometheus config with rule what I should to see:

  prometheus.rules: |-
    groups:
    - name: devopscube demo alert
      rules:
      - alert: High Pod Memory
        expr: up > 0
        for: 1m
        annotations:
          summary: High Memory Usage

  prometheus.yml: |-
    global:
      scrape_interval: 5s
      evaluation_interval: 5s
    rule_files:
      - /etc/prometheus/prometheus.rules
    alerting:
      alertmanagers:
      - scheme: http
        static_configs:
        - targets:
          - "alertmanager.newmonitoring.svc:9093"

alertmanager.yml
global:
resolve_timeout: 5m

route:
group_by: ['instance', 'severity']
group_wait: 30s
group_interval: 5m
repeat_interval: 30m
receiver: 'discord_webhook'
routes:
- matchers:
- instance =~ "x.x.x.x:5054|x.x.x.x:6060"
group_wait: 30s
receiver: 'discord_webhook'
continue: true

receivers:

• name: 'discord_webhook'
  webhook_configs:
  • url: 'http://nifty_gagarin:9094'
    send_resolved: true

Using docker image benjojo/alertmanager-discord:latest
Runs with Prometheus 2.24.0 and Grafana 7.3.6.

On alert getting such error:

2021/02/17 16:03:29 Listening on: 0.0.0.0:9094
2021/02/17 16:03:52 http: panic serving 10.0.8.4:57922: json: cannot unmarshal array into Go value of type main.alertManOut
goroutine 6 [running]:
net/http.(*conn).serve.func1(0xc00006ebe0)
        /usr/local/go/src/net/http/server.go:1801 +0x147
panic(0x6dd0c0, 0xc00004c230)
        /usr/local/go/src/runtime/panic.go:975 +0x47a
main.main.func1(0x79a560, 0xc0000b80e0, 0xc0000ac100)
        /go/src/mypackage/myapp/main.go:110 +0xd54
net/http.HandlerFunc.ServeHTTP(0xc000040c90, 0x79a560, 0xc0000b80e0, 0xc0000ac100)
        /usr/local/go/src/net/http/server.go:2042 +0x44
net/http.serverHandler.ServeHTTP(0xc0000b8000, 0x79a560, 0xc0000b80e0, 0xc0000ac100)
        /usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc00006ebe0, 0x79ab60, 0xc000010500)
        /usr/local/go/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2969 +0x36c
2021/02/17 16:03:52 http: panic serving 10.0.8.4:57924: json: cannot unmarshal array into Go value of type main.alertManOut

Hello,
i run the alert manager with docker
but I found this in Prometheus logs
prometheus[16658]: level=error ts=2021-01-18T20:15:47.759Z caller=notifier.go:524 component=notifier alertmanager=http://localhost:9094/api/v1/alerts count=1 msg="Error sending alert" err="Post http://localhost:9094/api/v1/alerts: EOF"

also I cannot make any get requests to this endpoint :
curl http://localhost:9094 curl: (52) Empty reply from server

and this is from container log
docker logs alertmanager -f 2021/01/18 20:40:52 Listening on: 0.0.0.0:9094 2021/01/18 20:40:58 http: panic serving 172.22.0.1:34486: unexpected end of JSON input goroutine 6 [running]: net/http.(*conn).serve.func1(0xc000076be0) /usr/local/go/src/net/http/server.go:1801 +0x147 panic(0x6dd060, 0xc00000c2e0) /usr/local/go/src/runtime/panic.go:975 +0x47a main.main.func1(0x79a560, 0xc0000c00e0, 0xc0000b4600) /go/src/mypackage/myapp/main.go:110 +0xd54 net/http.HandlerFunc.ServeHTTP(0xc000010cc0, 0x79a560, 0xc0000c00e0, 0xc0000b4600) /usr/local/go/src/net/http/server.go:2042 +0x44 net/http.serverHandler.ServeHTTP(0xc0000c0000, 0x79a560, 0xc0000c00e0, 0xc0000b4600) /usr/local/go/src/net/http/server.go:2843 +0xa3 net/http.(*conn).serve(0xc000076be0, 0x79ab60, 0xc000074480) /usr/local/go/src/net/http/server.go:1925 +0x8ad created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2969 +0x36c

Hello,

It would be cool if the CI can build the binary for each platform to avoid build on server (when you don't want to user docker).
It's really easy to setup on github CI with goreleaser I can create a pull request if you are okay with it.

Best regards,

Can this alertmanager work with prometheus default alert manager? I can see that the both share the same port.

I am attempting to use an external label, locID, with Prometheus alerts. I have the external label being sent to alertmanager-discord, but how does one configure what's going to discord; other then modifying main.go?

Here is the print-pretty formatted input JSON:

{
	"receiver": "prometheus-msteams",
	"status": "firing",
	"alerts": [{
		"status": "firing",
		"labels": {
			"alertname": "A test alert",
			"job": "kubernetes-apiservers",
			"locID": "00000000-0000-0000-0000-000000000000",
			"severity": "notice"
		},
		"annotations": {
			"description": "There are 1 kubernetes-apiservers jobs running",
			"summary": "This is only a test on locationID: "
		},
		"startsAt": "2020-08-31T18:36:25.881534544Z",
		"endsAt": "0001-01-01T00:00:00Z",
		"generatorURL": "http://prometheus-subscriber-server-6ccf748db-s4j9s:9090/graph?g0.expr=sum+by%28job%29+%28up%29+%3E+0\u0026g0.tab=1"
	}, {
		"status": "firing",
		"labels": {
			"alertname": "A test alert",
			"job": "kubernetes-nodes",
			"locID": "00000000-0000-0000-0000-000000000000",
			"severity": "notice"
		},
		"annotations": {
			"description": "There are 2 kubernetes-nodes jobs running",
			"summary": "This is only a test on locationID: "
		},
		"startsAt": "2020-08-31T18:35:25.881534544Z",
		"endsAt": "0001-01-01T00:00:00Z",
		"generatorURL": "http://prometheus-subscriber-server-6ccf748db-s4j9s:9090/graph?g0.expr=sum+by%28job%29+%28up%29+%3E+0\u0026g0.tab=1"
	}, {
		"status": "firing",
		"labels": {
			"alertname": "A test alert",
			"job": "kubernetes-nodes-cadvisor",
			"locID": "00000000-0000-0000-0000-000000000000",
			"severity": "notice"
		},
		"annotations": {
			"description": "There are 2 kubernetes-nodes-cadvisor jobs running",
			"summary": "This is only a test on locationID: "
		},
		"startsAt": "2020-08-31T18:35:25.881534544Z",
		"endsAt": "0001-01-01T00:00:00Z",
		"generatorURL": "http://prometheus-subscriber-server-6ccf748db-s4j9s:9090/graph?g0.expr=sum+by%28job%29+%28up%29+%3E+0\u0026g0.tab=1"
	}, {
		"status": "firing",
		"labels": {
			"alertname": "A test alert",
			"job": "kubernetes-service-endpoints",
			"locID": "00000000-0000-0000-0000-000000000000",
			"severity": "notice"
		},
		"annotations": {
			"description": "There are 1 kubernetes-service-endpoints jobs running",
			"summary": "This is only a test on locationID: "
		},
		"startsAt": "2020-08-31T18:35:25.881534544Z",
		"endsAt": "0001-01-01T00:00:00Z",
		"generatorURL": "http://prometheus-subscriber-server-6ccf748db-s4j9s:9090/graph?g0.expr=sum+by%28job%29+%28up%29+%3E+0\u0026g0.tab=1"
	}, {
		"status": "firing",
		"labels": {
			"alertname": "A test alert",
			"job": "prometheus",
			"locID": "00000000-0000-0000-0000-000000000000",
			"severity": "notice"
		},
		"annotations": {
			"description": "There are 1 prometheus jobs running",
			"summary": "This is only a test on locationID: "
		},
		"startsAt": "2020-08-31T18:36:25.881534544Z",
		"endsAt": "0001-01-01T00:00:00Z",
		"generatorURL": "http://prometheus-subscriber-server-6ccf748db-s4j9s:9090/graph?g0.expr=sum+by%28job%29+%28up%29+%3E+0\u0026g0.tab=1"
	}],
	"groupLabels": {
		"alertname": "A test alert"
	},
	"commonLabels": {
		"alertname": "A test alert",
		"locID": "00000000-0000-0000-0000-000000000000",
		"severity": "notice"
	},
	"commonAnnotations": {
		"summary": "This is only a test on locationID: "
	},
	"externalURL": "",
	"version": "4",
	"groupKey": "{}:{alertname=\"A test alert\"}"
}```