Comments (5)
This seems to coincide with intervals when head compaction happens. I think this process acquires a write lock and pending samples pile up in memory. @yeya24 do you see something similar in Cortex?
Just confirming the compactions happen at the same time as the memory spikes
![image](https://private-user-images.githubusercontent.com/11613073/318961080-6ceeb223-16d9-4a50-aec6-882b09445363.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTQzMDk2OTAsIm5iZiI6MTcxNDMwOTM5MCwicGF0aCI6Ii8xMTYxMzA3My8zMTg5NjEwODAtNmNlZWIyMjMtMTZkOS00YTUwLWFlYzYtODgyYjA5NDQ1MzYzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA0MjglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNDI4VDEzMDMxMFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTZhOTdhNGVlMmUzNTdmMDFkYzA4ZDBmNDcwYWMxMmFhOWQwZmJkMDNjODExOTg2ZWFkZDc5NTZjZmIxNTEyMDgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.TQW2kRQ9YtMzOke5NiZPQkZyXOuC4Y7B-O6QM6tqwZM)
from thanos.
This seems to coincide with intervals when head compaction happens. I think this process acquires a write lock and pending samples pile up in memory. @yeya24 do you see something similar in Cortex?
from thanos.
did you get context deadline exceeded (500) from ingestors during the WAL compaction?
from thanos.
Related Issues (20)
- Rate query failing from Grafana HOT 2
- Thanos receive context deadline exceeded: Permanent error: Post \"http://obs-thanos-receive.xxxsvc.cluster.local:19291/api/v1/receive\": context deadline exceeded HOT 2
- Thanos Store fills up the entire hard disk at startup HOT 3
- Support for encryption in transit for Memcached HOT 3
- Why does thanos query-frontend not support multiple downstream addresses?
- Thanos query fanout has high latency when a strict-store is unavailable HOT 3
- Ruler UI: `alert.query-template` is not honored inside the Rules UI HOT 4
- [Thanos Storegateway ]"failed to read index-header from disk; recreating" path=/data/01H1K45K0FRT36S1RCAWHW7R9A/index-heade HOT 1
- Adding User Agent to HTTP Logs
- Compact: Display TODO plan HOT 7
- compactor: does not compact 4 consecutive 2-hour blocks HOT 6
- compactor: series not 16-byte aligned error HOT 2
- Improved file access logging
- Sidecar: reporting as ready on startup when no Prometheus process is running
- tools bucket: Add ability to discover external labels from prometheus address for `upload-blocks` HOT 1
- Thanos Sidecar - Flush Endpoint HOT 9
- Grafana only shows raw data from Thanos HOT 4
- Instance Principal Provider - Region issue
- Consider X-Forwarded-For on HTTP/GRPC Logging
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from thanos.