Comments (4)
The accuracy of the record rule with regards to original data will depend on how often you evaluate the rule and will be impacted also by how often the data is sampled. Can you provide this information?
Keep in mind that these two queries are fundamentally different. They will almost never match 100%. For the raw query you have: how much that metric changed per minute. For the record, you have: a periodic evaluation of unknown interval (you didn't tell us how often you evaluate the rule) of how much that metric changed in the last minute.
You are querying slightly different series here. On the record query you have 48 series in the result, while in the raw data query you have only 4 series.
Also the time in both charts isn't aligned, so it becomes more difficult to look at discrepancies and think about why they happen.
from thanos.
@douglascamata The Thanos Ruler's eval-interval is set to 30s. Prometheus has a scrape_interval of 30s and is configured with remote_write to send data to Thanos Receive. Thanos Ruler queries data from Thanos Query, whose data store is Thanos Receive.
I tried setting the eval-interval to 1 minute, but there was no apparent change. However, when I changed the time range for the rate function from [1m] to [2m], the fluctuations were noticeably reduced.
from thanos.
The Thanos Ruler's eval-interval is set to 30s. Prometheus has a scrape_interval of 30s
This, paired with the fact that you are calculating the rate every 1m is not a good idea. A lot can happen because the periodicity of the Ruler and the Prometheus are not aligned (they don't happen at the same time) and your rate has a small enough interval that it might not have 2 points to calculate an actual rate.
My recommendation is to bump the eval interval and the rate interval to at least double or triple the scrape interval.
This is not a problem in Thanos.
from thanos.
Thank you for your help! I set the evaluation interval of the rule to 90 seconds and the rate interval to 2 minutes; it seems that increasing the rate interval has the most noticeable effect.
from thanos.
Related Issues (20)
- Thanos-store: Store Pod is not able to load certain blocks from object storage HOT 1
- query: Internal Server Error unknown targetHealth: "unknown" when trying to open the targets page
- Receive: high in flight requests and high context deadline exceeded and ingestion latency in main branch HOT 9
- Rate query failing from Grafana HOT 2
- receive: memory spikes during tenant WAL truncation HOT 7
- Thanos receive context deadline exceeded: Permanent error: Post \"http://obs-thanos-receive.xxxsvc.cluster.local:19291/api/v1/receive\": context deadline exceeded HOT 2
- Thanos Store fills up the entire hard disk at startup HOT 3
- Support for encryption in transit for Memcached HOT 3
- Why does thanos query-frontend not support multiple downstream addresses?
- Thanos query fanout has high latency when a strict-store is unavailable HOT 3
- Ruler UI: `alert.query-template` is not honored inside the Rules UI HOT 4
- [Thanos Storegateway ]"failed to read index-header from disk; recreating" path=/data/01H1K45K0FRT36S1RCAWHW7R9A/index-heade HOT 1
- Adding User Agent to HTTP Logs
- Compact: Display TODO plan HOT 7
- compactor: does not compact 4 consecutive 2-hour blocks HOT 6
- compactor: series not 16-byte aligned error HOT 2
- Improved file access logging
- Sidecar: reporting as ready on startup when no Prometheus process is running
- tools bucket: Add ability to discover external labels from prometheus address for `upload-blocks` HOT 1
- Thanos Sidecar - Flush Endpoint HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from thanos.