yatp could not preempt threads about agatedb HOT 6 OPEN

Seems that this is how thread pool is expected to work. It's more appropriate to use std::thread in agatedb, as we need long running threads instead of spawning short-running tasks.

from agatedb.

Can you explain more about "how thread pool is expected to work"? AgateDB should allow sharing global thread pool with other system.

from agatedb.

From yatp source code, it by default spawns a maximum of num_cpu threads. https://github.com/tikv/yatp/blob/master/src/pool/builder.rs#L37 Therefore, (from my understanding, if threads in thread pool won't be preempted), there would have at most num_cpu concurrently running threads in the background.

In agatedb with default configuration, we would have at least 5 concurrent tasks running in the background. If other systems have their thread pool misconfigured (like using num_cpus as maximum thread, for GitHub Action, it should be 2), then the database would easily go into stall state (e.g. compaction is never run).

from agatedb.

How about setting a larger capacity?

from agatedb.

How about setting a larger capacity?

If we have 5 threads, and the thread pool always has these 5 threads running, there seems no difference with std::thread :(

Anyway, I have put all threads back into yatp, and set max_threads for the pool. adf14f7

from agatedb.

Setting a larger capacity is to get around the CI issues. In fact, the thread pool should not be created by the library, it should be a builder parameter given by users. It is critical in resource limited environment to have more precise control on threads.

from agatedb.