sylr / alertmanager-splunkbot Goto Github PK

I recently upgraded to the latest prometheus-community https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack and the alertmanager webhook is unable to post to the splunkbot service. Following is the message reported in the alertmanger pod:
level=error ts=2020-12-01T19:04:34.242Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="splunk_webhook/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: http://alertmanager-splunkbot-service:44553"

Following is the config defined in the values.yaml for the splunk_webhook which works in the deprecated prometheus-operator

config:
global:
resolve_timeout: 5m
route:
group_by: [alertname]
group_wait: 30s
group_interval: 5m
repeat_interval: 1h
receiver: 'splunk_webhook'
routes:
- receiver: 'splunk_webhook'
continue: true

receivers:
- name: 'splunk_webhook'
  webhook_configs:
  - url: 'http://alertmanager-splunkbot-service:44553'
    send_resolved: true
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['instance']