Comments (5)
yeya24
commented on July 21, 2024
@BouchaaraAdil Have you checked your compactor metrics? Is it in halt status?
thanos_compact_halted you can find it via this metric.
https://thanos.io/tip/operating/compactor-backlog.md/#make-sure-compactors-are-running
from thanos.
BouchaaraAdil
commented on July 21, 2024
@yeya24 yes i already did the value is: 0
logs are mostly like:
caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=7m8.010456222s duration_ms=428010 cached=24894 returned=24894 partial=0
caller=fetcher.go:820 level=debug msg="removed replica label" label=prometheus_replica count=18750
caller=fetcher.go:407 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=60
caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=7m8.234159055s duration_ms=428234 cached=24894 returned=24894 partial=0
caller=blocks_cleaner.go:58 level=info msg="cleaning of blocks marked for deletion done"
caller=blocks_cleaner.go:44 level=info msg="started cleaning of blocks marked for deletion"
caller=clean.go:61 level=info msg="cleaning of aborted partial uploads done"
caller=clean.go:34 level=info msg="started cleaning of aborted partial uploads"
caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=8m35.588933685s duration_ms=515588 cached=24894 returned=24894 partial=0
caller=fetcher.go:820 level=debug msg="removed replica label" label=prometheus_replica count=18750
caller=fetcher.go:407 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=60
caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=8m35.392333486s duration_ms=515392 cached=24894 returned=24894 partial=0
caller=blocks_cleaner.go:58 level=info msg="cleaning of blocks marked for deletion done" caller=blocks_cleaner.go:44 level=info msg="started cleaning of blocks marked for deletion"
caller=clean.go:61 level=info msg="cleaning of aborted partial uploads done"
caller=clean.go:34 level=info msg="started cleaning of aborted partial uploads"
caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=8m11.753199423s duration_ms=491753 cached=24894 returned=24894 partial=0
caller=fetcher.go:820 level=debug msg="removed replica label" label=prometheus_replica count=18750
caller=fetcher.go:407 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=60
caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=8m11.674345163s duration_ms=491674 cached=24894 returned=24894 partial=0
caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=7m35.466654203s duration_ms=455466 cached=24894 returned=24894 partial=0
caller=fetcher.go:820 level=debug msg="removed replica label" label=prometheus_replica count=18750
caller=fetcher.go:407 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=60
caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=7m35.686672464s duration_ms=455686 cached=24894 returned=24894 partial=0
caller=blocks_cleaner.go:58 level=info msg="cleaning of blocks marked for deletion done"
caller=blocks_cleaner.go:44 level=info msg="started cleaning of blocks marked for deletion"
caller=clean.go:61 level=info msg="cleaning of aborted partial uploads done"
caller=clean.go:34 level=info msg="started cleaning of aborted partial uploads"
caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=6m37.443088828s duration_ms=397443 cached=24894 returned=24894 partial=0
caller=fetcher.go:820 level=debug msg="removed replica label" label=prometheus_replica count=18750
caller=fetcher.go:407 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=60
caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=6m36.845099529s duration_ms=396845 cached=24894 returned=24894 partial=0
caller=blocks_cleaner.go:58 level=info msg="cleaning of blocks marked for deletion done"
caller=blocks_cleaner.go:44 level=info msg="started cleaning of blocks marked for deletion"
caller=clean.go:61 level=info msg="cleaning of aborted partial uploads done"
caller=clean.go:34 level=info msg="started cleaning of aborted partial uploads"
caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=6m39.587074548s duration_ms=399587 cached=24894 returned=24894 partial=0
caller=fetcher.go:820 level=debug msg="removed replica label" label=prometheus_replica count=18750
caller=fetcher.go:407 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=60
caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=6m40.384243402s duration_ms=400384 cached=24894 returned=24894 partial=0
caller=fetcher.go:557 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=6m54.588252009s duration_ms=414588 cached=24894 returned=24791 partial=0
caller=fetcher.go:820 level=debug msg="removed replica label" label=prometheus_replica count=18647
caller=fetcher.go:867 level=debug msg="block is too fresh for now" block=01HSJP6XN8DPDBJ9XHZTTWDWRD
caller=fetcher.go:867 level=debug msg="block is too fresh for now" block=01HSJP6XKT2BVBNEB3THRGR0WC
caller=fetcher.go:867 level=debug msg="block is too fresh for now" block=01HSJP6X2G14XJ04WJ53CZXRKC
caller=fetcher.go:867 level=debug msg="block is too fresh for now" block=01HSJP6WVZ01H29HPT1QGK07T5
and the rate of compaction is very low, it starts only after recreating the compactor pod, when I removed the vertical compaction, the compaction rate was high and started to compact even fresher blocks of 2hours to 8,6 hours
from thanos.
Related Issues (20)
- bucketweb: Show block size in details pane HOT 4
- [Store] slice bounds out of range [:1029] with capacity 1024 HOT 4
- Thanos ruler: Alertmanager v0.27.0 /api/v1 endpoints deprecated HOT 2
- receive-distributor: always restart, SIGSEGV and panic HOT 3
- Thanos Ruler: The record and the result of its expression calculated in Thanos Query show significant differences. HOT 4
- prometheus-thanos-sidecar helm chart has incorrect volume mounts, pods error on startup HOT 2
- 0.32.0 caused spike in network traffic HOT 8
- bucketweb: attempting to view overlapping blocks results in error page HOT 1
- store: redis client cache-size >0 results startup error
- Automatic detection of memory limits and configure GOMEMLIMIT to match.
- Thanos compactor "--min-time" parameter not working HOT 3
- ui: Ability to disable specific query warnings from being displayed
- Tracing: Add missing sampler types HOT 1
- Thanos sidecar doesn't upload full blocks HOT 12
- Blocks UI: Find sources for external labels HOT 8
- Thanos-store: Store Pod is not able to load certain blocks from object storage HOT 1
- query: Internal Server Error unknown targetHealth: "unknown" when trying to open the targets page
- Receive: high in flight requests and high context deadline exceeded and ingestion latency in main branch HOT 9
- Rate query failing from Grafana HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from thanos.