observatorium / thanos-rule-syncer Goto Github PK

Problem

Thanos Rule Syncer currently supports fetching rules for a single tenant from Observatorium API. This means that if you are running Obs API with multiple tenants (almost always), you'd need to run one Thanos Rule Syncer sidecar for each tenant.

Proposed Solution

We can add support for specifying multiple tenants and their credentials as input to Rule Syncer (maybe as a yaml file) and then we can fetch rules for all of them, combine in a single file, and then follow the same pattern we followed for a single tenant.

As mentioned https://observatorium.io/docs/proposals/20201019-prometheus-rules-in-observatorium-api.md/#2-sync-rules-to-rule, after thanos-rule-syncer gets data from observatorium/Rules API, it needs to:

• combine the data into one file that it's shared with Thanos Rule
• calls Thanos Rule's /reload endpoint to reload the config
• Thanos Rule then starts to evaluate the recording/alerting rules and stores the results in a local TSDB instance