Comments (3)
Hey, indeed I did some changes on the eagerRespSet
that seems to be the culprit here. IIRC, I removed the check on the context there because it caused a race condition between it and the serverAsClient (implementation of the storepb.Store_SeriesClient
interface) in which the eagerRespSet
or lazyRespSet
would stop calling the Recv()
too early, leaving a dangling goroutine.
I would suspect a bad context management
Looking quickly at the code, blockSeriesClient
seems to be used by the querier, and in the Recv(), the context is not checked directly but it seems to used in the nextBatch
function. There might be a code path where a done context is silently ignored 🤷 . I need to have a deeper look at it. This is just an hypothesis.
from thanos.
I have taken some goroutine dumps from our queriers.
goroutine dump from one of our thanos-querier pods
goroutine dump from one of our thanos-querier-tls pods
We also see a correlation between memory usage and goroutines, shown in these dashboards
from thanos.
Maybe related to #6948?
cc @thibaultmg , any chance there could be something missing when chaining queriers?
from thanos.
Related Issues (20)
- Doc for out of order HOT 1
- sidecar: Do not crash when Object Storage is not accessible HOT 1
- The result of "Use Deduplication" is not as expected HOT 9
- Receiver samples limit should include native histograms
- Docs version dropdown overflows the page
- Rule: Support QueryOffset
- Add support to the Info field in the body of query API responses
- Expose more rueidis clientOpts
- Index Cache: Change fetchMulti interface to return slice rather than map HOT 3
- Sidecar: After a block upload was delayed, all subsequent block uploads were delayed as well HOT 4
- Can we filter out the blocks that are already fully compressed and then downsample them. This way downsampled blocks will not be compressed.There shouldn't be any other problems with this, right? HOT 1
- query/sidecar: same store.limits flags on both components
- Thanos update causing compactor to err on sync before retention HOT 3
- query: regression in v0.36.0 - concurrent_gate_queries periodically reaching limit HOT 2
- Thanos querier rate/increase function creating huge spikes/incorrect results when deduplication is enabled
- Make TLS versions and ciphers configurable HOT 1
- FT add feature flag for 'promql-experimental-functions'. HOT 3
- Improvement of OpenSSF Scorecard Score HOT 1
- Thanos Query: gaps in deduplicated data
- Segmentation Violation when connecting to Kafka
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from thanos.