compactor: Maximizing CPU usage about thanos HOT 7 CLOSED

Also noticing that most of my compactors are spending time in cleaning of aborted partial uploads, and cleaning of blocks marked for deletion. The latter makes sense but not sure of the former.

They keep on switching between this deletion process and compaction.

from thanos.

I had the impression that if I set a concurrency value equal to the no. of cores assigned, I should see a very high % of CPU used but instead, even with ~16 cores assigned and --compact.concurrency=16, % CPU used was still hovering around 10%.

Compaction might not be always happening. It wait for you have enough data to be compacted. Even if you specify 16 concurrency to compact blocks, it doesn't mean you always have 16 compaction jobs to run. You probably only have 1 then only 1 core is used. We don't support using more than 1 core within a single compaction job because it is now single-threaded.
The CPU usage pattern is like mostly idle -> CPU high during compaction time -> idle.

Before it actually compacts blocks, compactor might spend quite long time downloading required blocks and analyzing the index file so you might see CPU usage still low here as it is IO intensive.

from thanos.

You probably only have 1 then only 1 core is used. We don't support using more than 1 core within a single compaction job because it is now single-threaded.

Got it, that clears my confusion.

What's the definition of compaction job here? And in what scenario does more concurrency come into effect?

from thanos.

What's the definition of compaction job here?

A single compaction which produces 1 output block.

And in what scenario does more concurrency come into effect?

I can image if you have multiple clusters with different cluster labels, then multiple compaction jobs will be available since each cluster (they should have their own ext labels) will have its own compaction job at the same time.

Within a single compaction group, there might be multiple compaction jobs available (imaging you have a huge compaction backlog), but we only support 1 concurrency per group. This is a limitation in Thanos right now.

from thanos.

I can image if you have multiple clusters with different cluster labels, then multiple compaction jobs will be available since each cluster (they should have their own ext labels) will have its own compaction job at the same time.

I see. Is this configurable or does it only recognize the cluster label? Can it be configured to run different jobs for set of blocks differentiated via the prometheus label (cluster value is the same)?

Within a single compaction group, there might be multiple compaction jobs available (imaging you have a huge compaction backlog), but we only support 1 concurrency per group. This is a limitation in Thanos right now.

Ah that makes sense. Thanks for clarifying.

from thanos.

I see. Is this configurable or does it only recognize the cluster label? Can it be configured to run different jobs for set of blocks differentiated via the prometheus label (cluster value is the same)?

It is just different ext labels you configured. Doesn't have to be cluster label.

from thanos.

I will convert this into a discussion.

from thanos.