Configuring prometheus to use a different alertmanager about prometheus-boshrelease HOT 8 CLOSED

Hi @pcfadminx

I think the you want to use the following spec:

https://github.com/bosh-prometheus/prometheus-boshrelease/blob/66868a788deb46f6ebe7cbe9482c55a09e120980/jobs/prometheus2/spec#L55

to define your alertmanager config.
Let me suggest you an adjusted version of your ips file.

- type: replace
  path: /instance_groups/name=prometheus2/jobs/name=prometheus2/properties?/prometheus/alertmanagers/-
  value:
    static_configs:
    - targets:
      - https://my_standalone_alertmanager:9093

That means your path was correct until the properties, after the properties you basically add the spec you want to adjust, in your case the spec is called prometheus.alertmanagers so you add that after the properties to your path and you are fine.

from prometheus-boshrelease.

@benjaminguttmann-avtq

I made a deployment using that ops file. I can see the new items in the manifest in bosh. But, alerts are still going to the default alertmanager included in bosh-prometheus.

The relevant manifest section of it currently appears like this:

  jobs:
  - name: prometheus2
    properties:
      prometheus:
        alertmanagers:
        - static_configs:
          - targets:
            - http://my_standalone_alertmanager:9093

I'm thinking maybe the name of the property items or the order they are in or something like that is wrong, but it's hard to say as I don't have a good config to reference/compare.

The end goal will be to send alerts to multiple different alert managers at the same time. If you know of any other way to achieve this, please let me know.

Thanks

from prometheus-boshrelease.

In addition to the mentioned spec, the configuration also uses bosh links.

https://github.com/bosh-prometheus/prometheus-boshrelease/blob/66868a788deb46f6ebe7cbe9482c55a09e120980/jobs/prometheus2/spec#L21-L23

Do you deploy the the default alertmanager included in prometheus deployment ? If so, you can either:

• delete the alert manager from the deployment:

- type: remove
  path: /instance_groups/name=alertmanager

• explicitly remove the generated link

- type: replace
  path: /instance_groups/name=prometheus2/jobs/name=prometheus2/consumes?/alertmanager
  value: nil

Below the piece of template that generates the actual configuration:

https://github.com/bosh-prometheus/prometheus-boshrelease/blob/66868a788deb46f6ebe7cbe9482c55a09e120980/jobs/prometheus2/templates/config/prometheus.yml#L41-L52

from prometheus-boshrelease.

@psycofdj

When I deploy it without the included alertmanager (plus with the ops file from a few posts above) prometheus doesn't connect to any alertmanager at all. I confirmed this by checking the config inside the prometheus instance - the section where it points to an alertmanager shows up blank.

I can achieve pointing to another alertmanager manually by SSHing into prometheus and editing configs, but need a way to do this through the manifest.

from prometheus-boshrelease.

The linked spec is new in latest v26.7.0 and I assume you're running a previous version of this bosh release.

In prior versions, you can use manual bosh link configuration as follow:

# this is manual bosh link configuration 
- type: replace
  path: /instance_groups/name=prometheus2/jobs/name=prometheus2/consumes?
  value:
    alertmanager:
      instances:
      - address: first.alertmanager.domain.org
      - address: second.alertmanager.domain.org
      properties:
        alertmanager:
          web:
            port: 9093

This approach is also compatible with the latest version v26.7.0.

from prometheus-boshrelease.

Update: I was able to get it to work on 26.7.0

But I can't get addresses with paths to work. How would I pass an address that includes paths? e.g. second.alertmanager.domain.org/some/path

When there's a path, it errors out saying the hostname is invalid.

from prometheus-boshrelease.

  jobs:
  - name: prometheus2
    properties:
      prometheus:
        alertmanagers:
        - 
          path_prefix: /to/my/path
          static_configs:
          -
            targets:
            - first.alertmanager.domain.org:<port>
            - second.alertmanager.domain.org:<port>

The rest is more a matter of pure prometheus config: see here: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#alertmanager_config

from prometheus-boshrelease.

I modified my ops file something like this and it seems to be working. Thanks!

- type: replace
  path: /instance_groups/name=prometheus2/jobs/name=prometheus2/properties?/prometheus/alertmanagers/-
  value:
    path_prefix: /to/my/path
    static_configs:
      - targets: ["second.alertmanager.domain.org:9093"]

from prometheus-boshrelease.