Ruler: Implement flag for max-source-resolution in the rule query about thanos HOT 7 OPEN

Working on this, will try to come with a PR.

from thanos.

@KingRebo38 What's the query are you running in Ruler?
We are thinking about detecting the resolution rather than setting up the flag manually.
A ruler instance level flag might not work well if you have other rules which don't need 5m resolution

from thanos.

@yeya24 The Queries are something like this:

rules:
  - expr: increase(kong_http_requests_total[14d])
    record: kong_http_requests_total:rate:14d
  - expr: increase(kong_http_requests_total[30d])
    record: kong_http_requests_total:rate:30d
    ...
    and so on

I also think the instance level flag might cause some issues, but for now I would solve it by sharding the ruler instances.

from thanos.

@KingRebo38 you can optimize these rules by reusing data from others.

rules:
  - expr: increase(kong_http_requests_total[1d])
    record: kong_http_requests_total:rate:1d
  - expr: sum_over_time(kong_http_requests_total:rate:1d[14d]) # sum the 1d-rate over 14d
    record: kong_http_requests_total:rate:14d
  - expr: sum_over_time(kong_http_requests_total:rate:1d[28d]) # sum the 1d-rate over 28d
    record: kong_http_requests_total:rate:28d

With this you basically have your own downsampling. As long as they are part of the same rule group, evaluation order is respected.

Edit: Ah, if your raw resolution retention is 7 days it won't work :(

from thanos.

@KingRebo38 does it work if you set max resource resolution to auto?

from thanos.

@douglascamata If I set it to auto, it defaults to the raw data

from thanos.

You can try then setting --query-range.request-downsampled on your Query Frontend.

from thanos.