[Bug]: [benchmark][cluster] L0 compaction task continues to rise, compression task may be stuck about milvus HOT 13 CLOSED

I guess the blocking here is the segment you have here, the more upload operations are.

Once you have 10000 segments, each upload takes 20ms, then, it's 200s.

Let's take a pool with cpu core * 8 nodes for concurrent file flush.

from milvus.

/unassign

from milvus.

It's not stucked, L0 compaction executes are slower and slower when segment num increases. And all the l0 tasks are pending to execute. Making MixCompaction unable to schedule.

need to refine scheduler to avoid this situation.

from milvus.

L0 compaction becomes slower because the bf?

from milvus.

maybe we need an extra stage of L0 compaction:

1. concurrently load bf into a datanode(maybe cached)
2. concurrently split all L0 segments to existing segments
3. lock the L1/L2 compaction
4. copy delta logs if necessary to compacted result
5. release the lock.

I would assume this will simply release the pressure of compaction and reduce the lock holding time

from milvus.

we may also need a cache mechanism for cache BF for fast loading in the future.

maybe cache it on log node is good idea? @XuanYang-cn @congqixia @tedxu

from milvus.

L0 compaction becomes slower because the bf?

@xiaofan-luan
L0 rely on the cached bf in datanode, becase we want datanode to work when disable L0. So currently, L0 compaction process doesn't need to load bf from S3, there're all in datanode already.

Need tracing to know the time costs, as segment num increased to 8000, l0 compaction p99 cost 15mins.

And l0 compaction tasks by pass the scheduler max task num, submitted hundreds of L0 tasks. causing no MixCompaction can be scheduled. And no MixCompaction can be scheduled is the root cause of the increasing segment num.

#31270 would prevent scheduler to append endless l0 tasks.

from milvus.

@wangting0128 please help verify, also could you enable tracing during tests?
/assign @wangting0128

from milvus.

A typcial batch L0 process tracing

from milvus.

Very obvious and quick enahncement: change linear upload to batch upload

from milvus.

make sense to me

from milvus.

After verification, under the current mechanism, there are two ways to alleviate the problem of slow compaction speed under concurrent DML.

verified image：2.4-20240318-506534c2

1. Add dataNode count

2. Adjust config parameters to improve compaction concurrency

    dataCoord:
      compaction:
        workerMaxParallelTaskNum: 5
        maxParallelTaskNum: 20

from milvus.

After verification, under the current mechanism, there are two ways to alleviate the problem of slow compaction speed under concurrent DML.

verified image：2.4-20240318-506534c2

1. Add dataNode count

2. Adjust config parameters to improve compaction concurrency

    dataCoord:
      compaction:
        workerMaxParallelTaskNum: 5
        maxParallelTaskNum: 20

After verification, this issue can be closed

from milvus.