Comments (4)
@stalemate3 you are telling Grafana that you want the minimum step to be 1 hour. This means one datapoint per hour at minimum... at this step size Thanos decided that it will use raw data to answer the query. By default, Thanos does not reply to queries mixing data from different levels -- only a single is used to answer queries. So this is why you don't see the downsampled data there.
You can do two things:
- Enable add the CLI arg
--query-range.request-downsampled=true
to your Query Frontend, if you run it. This makes Thanos query for all the downsampling levels, going up in resolution one by one, to check if there's data. - Remove or change that min step in your query to try to hit an specific downsampling levels.
Personally I would go with option 1.
from thanos.
Sorry for the late reply and thanks for the suggestions, there is some progress with my issue. I've tried both of your suggestions, the 1st one doesn't seem to do anything when I redeploy Query Frontend with that arg from Grafana side. For the second one, that's only there because of my desperation to try random steps to magically fix my problem, sorry about that.
From another issue here I set the --query.auto-downsampling
to Query and as I've played around with the min step I set it to 5h and voila:
Anything below that is going to give back the same data as in my original post. I'm not sure why 5h is this magic spot but now there are these gaps in the data. If I set it above 5h the gap widens. Any ideas to fix these gaps?
Thanks for the ideas, I really appreciate it!
from thanos.
I think there is some heuristic that max resolution is step/5 or something somewhere! 5h would be answered from 1h down samples data then
from thanos.
I face the same problem with a similar setup (just running it on on-prem K8s and using the latest Thanos and the latest Bitnami Helm chart). For some reason i don't see in Grafana down-sampled data even if the query component has --query.auto-downsampling
set. When i go to Thanos-frontend and choose Auto downsampling i can see the metrics without any issue. If i use custom parameter max_source_resolution=3600000
for my thanos datasource in Grafana ot set the "Min step" to 5h like @stalemate3, i can see the metrics also there. Anyone any idea?
from thanos.
Related Issues (20)
- thanos compact panic on GetActiveAndPartialBlockIDs HOT 1
- why can't thanos fail properly on errors (like everone else)? HOT 1
- compactor: Fails cleaning blocks marked for deletion with Access Denied (solved)
- Caching Bucket File Unknown Flag HOT 2
- compactor: add series: symbol table size exceeds X bytes: X HOT 3
- Enhance Thanos Ruler UI with Deep Linking Capabilities for Specific Alerts
- Thanos Store Does Not Reflect S3 Storage Unavailability in Health Checks HOT 2
- fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?) HOT 1
- Compactor Warning: block has no labels left, creating one
- Compactor: Deadlock on S3 error during meta sync HOT 3
- Problem with receiver and store HOT 1
- Compactor: Skip compaction block index verification if metadata file hash enabled
- Compactor: Context canceled HOT 2
- Ruler: Evaluations stop without errors or logs
- Receiver: Introduce max bucket limit for native histograms
- Enable OpenSSF Scorecard to enhance security practices across the project
- Compactor: UI shows `[object Object]`
- Check correct error when opening rule files in tools
- thanos-sidecar uploads OOO block before Prometheus compacts it, upsetting thanos-compact HOT 1
- Query: possible goroutine leak seen between queriers on v0.35.1 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from thanos.