Comments (4)
The query in the UI uses [1m] ~ from the sample it feels like you have 30s scrape frequency. Does it also happen with [5m]? I wrote a qucik test with your given inputs and the result series looks somewhat like:
samples: []sample{
{t: 1713668216000, f: 9389.87},
{t: 1713668224000, f: 9389.94},
{t: 1713668254000, f: 9389.95},
{t: 1713668284000, f: 9390.02},
{t: 1713668314000, f: 9390.03},
{t: 1713668344000, f: 9390.33},
{t: 1713668374000, f: 9390.83},
{t: 1713668404000, f: 9391.13},
{t: 1713668434000, f: 9391.38},
{t: 1713668464000, f: 9391.61},
{t: 1713668494000, f: 9393.53}},
},
from thanos.
It looks like all samples are there and do have proper 30s scrape interval between them; it could be that your 1m windows are aligned in a way that only one sample is contained in the window which would break rate. I think this is an issue with too small window, but the deduplication result looks somewhat correct to me except that we have one sample too much at the beginning
from thanos.
We have evidence about dedup logic bug as well, here are the proof:
First graph is data points missing when getting the results from thanos receiver with replicationFactor == 3
, and because we are rolling update the receiver pods, so 1 copy was absent for sure, however after check Deduplication, it still have dips:
After the data got compact and returned from store, the results become correct with no dips:
from thanos.
seems related to this issue: #981
from thanos.
Related Issues (20)
- Max and min pointed at Sidecars not working on 0.35 HOT 15
- `ThanosSidecarBucketOperationsFailed` alert is flaky
- PR Title Validation HOT 3
- Thanos Receive Pod is crashing with Readiness and livness Probe Failed
- Thanos ruler vs. eventual consistency of metrics
- Can Huawei's OBS storage be supported? HOT 2
- Thanos React-app : Proxy server for thanos-query
- Query: update of endpoint failed...context deadline exceeded
- Thanos Chart 0.34.0 app version 12.23.1
- Thanos receive fails "no space left on device"
- sidecar: Greatly increased Thanos sidecar memory usage from 0.32.2 to 0.32.3, still exists in 0.35.0 HOT 3
- api/v1/label returns wrong values HOT 3
- Regression in thanos v0.35.1 HOT 3
- Thanos Receiver: Router/Ingestor setup no longer returns `thanos_receive_write_timeseries_*` and `thanos_receive_write_samples_*` metrics with thanos v0.35.1 HOT 3
- Extend Thanos bucket rewrite to support filtered archiving of existing blocks
- Support additional aggregates for downsampling
- Store Announced LabelSets Unexpected
- Warning in Grafana 11+: Thanos Receive dashboard depends on Angular
- compactor: Thanos Compactor compaction level number being set to 93+ for 1 half of HA Prometheus (S3 Storage) HOT 3
- Query: Network bandwidth usage upward of 500MB/s between Querier and configured stores HOT 17
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from thanos.