tikv / agatedb Goto Github PK

View Code? Open in Web Editor NEW

818.0 34.0 75.0 503 KB

A persistent key-value storage in rust.

License: Apache License 2.0

Rust 99.84% Batchfile 0.01% Makefile 0.14%

agatedb's Issues

Remove temporary directories in test

This can leave dirty file in temporary directory on Windows.

Originally posted by @BusyJay in #26 (comment)

Feature Request: Adaptive Radix Tree memtable

As https://db.in.tum.de/~leis/papers/ART.pdf showed, the adaptive radix tree is mush more faster than other comparable search tree/list.

Feature Request: async IO (io_uring on cloud block storage)

Async IO is an effective mechanism to improve the IO utilization by using few threads. When comes to cloud disk, aka cloud block storage, the IO latency is higher than local disk, if using synchronous read/write in that situation, threads will be blocked longer. And this will lower the CPU utilization. So async IO is required more necessary when come to cloud disk.

https://kernel.dk/io_uring.pdf io_uring is a comprehensive async IO interface of Linux, AgateDB can introduce the async IO mechanism powered by lower-level io_uring.

(clippy::uninit_assumed_init): this call for this type may be undefined behavior

agatedb/src/value.rs

Line 129 in 4aa6420

let mut arr: [u8; 12] = unsafe { MaybeUninit::uninit().assume_init() };

should change to

let mut arr: [MaybeUninit<u8>; 12] = unsafe { MaybeUninit::uninit().assume_init() };

let mut arr = [0u8; 12];

error: this call for this type may be undefined behavior
   --> src/value.rs:129:42
    |
129 |         let mut arr: [u8; 12] = unsafe { MaybeUninit::uninit().assume_init() };
    |                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: `#[deny(clippy::uninit_assumed_init)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#uninit_assumed_init

为什么db.update中set的key-value不生效?

运行程序 https://github.com/rmw-lib/agatedb/tree/cli/examples

    db.update(|tx| {
        use agatedb::{key_with_ts, util::unix_time};
        use bytes::BytesMut;
        let key = key_with_ts(BytesMut::from("3"), unix_time());
        tx.set(key, "2".into())
    })?;

然后读取，读取不到

但是这样写入就可以读取到

    pub fn set(&self, key: impl Into<BytesMut>, value: impl Into<Bytes>) -> Result<()> {
        use crate::util::unix_time;
        let key = key_with_ts(key.into(), unix_time());
        let req = Request {
            entries: vec![Entry::new(key, value.into())],
            ptrs: vec![],
            done: None,
        };
        self.write_to_lsm(req)
    }

get函数定义如下

    pub fn get(&self, key: impl Into<BytesMut>) -> Result<Value> {
        let key = key_with_ts(key.into(), std::u64::MAX);
        self.core.get(&key)
    }

运行演示截图如下

write_to_lsm我感觉是一个写入磁盘操作，是不是Transaction执行之后也需要手工运行个什么来写入磁盘？难道数据库被drop的时候不会自动写入磁盘吗？

我记得redis和rocksdb都一个设置，可以配置落盘的周期，比如让程序每3秒钟落盘一次，有这样的配置吗？

Running tests with code coverage is endless after switch compaction on

#148 start the background compaction threads, but seems this will cause the "Running Tests With Code Coverage" endless.

Proposal: a better structure for LevelHandler

Problem

LevelHandler will re-sort the whole sst file in one level. But if there are 100K files in one level, once sort operation may cost 20~100ms (I bench it in my macbook 2020). And if there are many compaction jobs running together, they may block the read thread too long.

Solution

I propose a two-level b+ tree for LevelHandler and it will split the sst into much page. Every time it changed it only copy the origin page and update one row of it, and then replace it in a short time with holding mutex. I think two-level is enough because sort one thousand string can be finished in 1ms.

Feature Request: bi-direction skiplist

SkipList is a single direction skiplist, it is ok when we retrieve values in ascend order, but it is slow when we call prev to scan the skiplist in descend order.

Descend reading usually happens when we want to read latest records run query like SELECT * FROM t ORDER BY time DESC [LIMIT n] in SQL database. This is a very common scenario. So it is necessary for AgateDB to introduce the bi-direction skiplist to improve the descend reading performance.

一个我搞不定的错误 No such file or directory (os error 2)

代码见我的fork的cli分支 https://github.com/rmw-lib/agatedb/tree/cli/examples

我的想法是给 agatedb 写一个命令行工具，也可以当做学习agatedb的demo代码

代码是在 #178 (simple agatedb get put) 基础上继续开发的

但是目前卡在了这个错误上 No such file or directory (os error 2)，因为这个错误没代码行提示，我也搞不清楚为啥出错

截图如下

另外，每次运行 /tmp/adb 下面都会多一个 .mem文件，感觉这也不太正常

Read lock on LevelHandler while get

When getting value from LSM, agatedb iters on levelhandlers and calls LevelHandler::get() under a RwLockReadGuard which will cause the whole read I/O path on this level under a read lock. It is always not a good idea to do I/O under lock even it is a read lock.

See:

agatedb/src/levels.rs

Line 931 in 475e17b

let v = handler.read()?.get(key);

I think such read lock is only necessary for fetching associated Tables from LevelHandler and the exact read work can be done outside the lock.

This logic is same as what it is in badger.
https://github.com/dgraph-io/badger/blob/7d159dd923ac9e470ffa76365f9b29ccbd038b0d/levels.go#L1602
The comment says it will call h.RLock() and h.RUnlock(), but exactly they are called in getTableForKey() which only locked when getting tables.
https://github.com/dgraph-io/badger/blob/master/level_handler.go#L241-L243

BTW, as Table might be removed (I/O) when the struct drop and this might happened when modifying LevelHandler under write lock. So this might influence read? I' mot sure whether it's right?

Implement Storage Engine

Development Task

Implement Storage Engine, R/W API and Transaction API (3-4 weeks)

Development progress

This section tracks what has been done (on develop branch)

Pull Requests

This section tracks progress on master branch

Feature Request: get key only

Some times we just want to check if a key is existed in the DB, for example when we call transaction put to insert a row to TiKV we need to check if the key exist(in TiKV we call it write conflict check).

Currently we use get to check if the key exist, but the get will also return the value which will cost extra memcpy. If we can add a key only read option for get, we can eliminate the memcpy of value part.

The skiplist is not concurrent but could be used concurrently

The arena in skiplist does not support concurrent operations(#116), but it could be used concurrently when allow_concurrent_write is set to true.

Concurrent tests such as bench_read_write_skiplist will crash with [SIGSEGV](signal: 11, SIGSEGV: invalid memory reference) when reducing the arena capacity, for example changing the https://github.com/tikv/agatedb/blob/master/skiplist/benches/bench.rs#L47 size from 512 << 20 to 512.

Supporting the concurrent arena allocator type may be necessary.

Implement Basic Structures

Development Task

Implement basic data structures (2-3 weeks)

Development Progress

main tasks
- add CI integration
- switch to prost
- table builder
- SST and iterator
- SST checksum
- SST and iterator benchmark
- bloom filter
- ~~cache (seems not necessary)~~
- ~~vLog (requires integration with GC, only implement basic struct in this stage)~~
bug fixes and patches
- enable clippy and enable warn_unused
- fix skiplist benchmark failures
- fix test timeout failures
- add sanitizer #10
- use jemalloc in benchmark to avoid fluctuations
improvements
- use mmapfile instead of file+vec to improve read throughput
- auto calculate length, and eliminate the need to store len along with Bytes or Vec
- implement some memory copy optimization
- use Result<()> instead of iter.valid()

Pull Requests

yatp could not preempt threads

After introducing MVCC and txn, we now have 3 long-running background threads.

memtable flush
batching write request
compaction * 3

If we put all threads in one thread pool, one of them won't even begin to execute in GitHub Action. Current workaround is to use std::thread for flush and write, and pool for compaction. We may later investigate how to integrate all threads into a thread pool.

Benchmark

Development Task

Do benchmarks on agatedb (1-2 weeks)

Development Progress

micro benchmark with badger-bench
- plan
- bench
~~macro benchmark with TiKV~~

[dev] ts cannot be 0

Current logic makes key with timestamp 0 not accessible. Will investigate into timestamp-related logic.

[dev] Tracking issue for database persistence

Currently, we simply delete table files after we drop the last reference Arc to TableInner. This causes several issues.

We had to forget (leak) table manually, thus causing leak sanitizer complain. https://github.com/tikv/agatedb/pull/60/checks?check_run_id=1452618735
By using mem::forget, we also have to manually close underlying file objects.

Proposal: Ingest External SST

This issue describe a possible way to ingest external SST file to a running agatedb instance just as rocksdb.

Background

TiKV use ingest external SST feature for some use case. To integrate agatedb into TiKV, we need this feature.

From Creating and Ingesting SST files · facebook/rocksdb Wiki (github.com), ingesting a SST needs follow steps:

copy or link the file(s) into the DB directory
block (not skip) writes to the DB and assign correct sequence number to ingested file
flush memtable if range overlap (SST file's key range overlap with memtable)
assign the file to LSM-tree

Things go different in agatedb.

Analysis

Step 1 and step 4 are the same.

For step 3, it is unnecessary to flush memtable when overlap for agatedb. Agatedb reads all levels when get and picks the latest one (less or equal to read_ts), so it's okay to have new keys below old keys in LSM-tree. This really makes the whole process efficient. (Please point out if I'm wrong)

For step 2, it is the most difficult: How to protect ACID during and after ingestion?

Agatedb has ACID transaction (SSI). The architecture is totally different from rocksdb. We have a read_ts and a commit_ts and we have conflict check when commit.

Possible Implementation

Main idea: Regard ingesting SST files as writing large range of keys and make the whole process a transaction.

Ingesting files shall be an expensive job.

Time Point: before commit

Add files, fetch new file id from LevelsController then move or copy files to DB dir with allocated file id(name).
Verify file checksum (optional) and get file infomation (just open Table?).
Make a new transaction, mark update as true (add a ingest flag for transaction?).
(TBD) other checks…

Check files before starting a txn.

Time Point: commit

Same as current, but add ingest range field to CommittedTxn in CommitInfo which contains smallest and largest keys in each ingest files.
This is for fast but inaccurate conflict check, otherwise we need to calc every key hash in SST which really takes time.
Set commit_ts as Table's global_version
Send to write channel as usual (see below)
Process Ingest, find suitable level for each files. (Which will take write lock of LevelHandler , and will block read process of other txn)
Write manifest file. The file's global_version is stored in manifest.

A question here. I see we protect the order of Requests send to write channel the same as commit_ts and I wonder why? For WAL in order?

If ingest job can be out-of-order, I think it's possible and better to make ingest job running in current thread and not sending to write channel.

You can see that the whole commit process has only one small I/O — append to manifest. But when picking level, it takes time if there are many tables in this level and this happens under a write lock of LevelHandler .

Conflict Check

There are two ways for conflict check as I described in TP: commit (step 1) .

Calculate every key's hash in ingested files and then conflict check works as same right now. For performance problem, we can calculate key hash before transaction.
Add ingest range field to CommittedTxn in CommitInfo .
And for any update txn, when adding read keys, mark smallest and largest read key for this txn. When checking conflict, beside checking hash, also check if read range overlap with ingest range.
This is really inaccurate and will cause many txn conflict but I think it makes sense.

The second way needs to refactor transaction much more.

(A more simple way is to break ACID…hhh)

Global Version

This concept is the same as rocksdb. For ingested files, all keys has wrong version and it's unreasonable to modify each. When file has a global version, every key in this file has this version.

In latest rocksdb, this value is stored in manifest to avoid random write to ingested files.

We need to add a field in meta.proto and update Table iterator (also block iterator) to fit this feature.

global_version may have ambiguity. Maybe another name when impl.

Discuss

A new transaction struct only for ingest or refactor current?
Which way to check conflict? (or any better idea?)
Send to write channel or done in current thread?
Any good idea to impliment this feature?
Strategy to pick level. I think from the bottom to check if there is overlap is nice.
What if external files (in one job) has overlap range? Rocksdb will put all files at level0.

[Question] Is this still being developed?

I am looking for an embedded KV store in Rust and found this crate.
Just want to know the status of this work.

Issue template is confusing

As organization template is used by default, the create issue page is really confusing. We could later copy TiKV issue template here.

[dev] compaction deadlock

compact #1 on level 0
5976
error when fill L0 to Lbase CustomError("KeyRange { left: b\"00000e2e\\0\\0\\0\\0\\0\\0\\0\\0\", right: b\"00000eb3\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\", inf: false } overlap with this level [KeyRange { left: b\"\", right: b\"\", inf: true }]")
5977
error while compaction: CustomError("only compactor zero can compact L0 to L0")
5978
compact #1 on level 0
5979
error when fill L0 to Lbase CustomError("KeyRange { left: b\"00000e2e\\0\\0\\0\\0\\0\\0\\0\\0\", right: b\"00000eb3\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\", inf: false } overlap with this level [KeyRange { left: b\"\", right: b\"\", inf: true }]")

https://github.com/tikv/agatedb/pull/73/checks?check_run_id=1484484527

Compile error: no method named `steal_batch_with_limit_and_pop` found for struct `Injector` in the current scope

➜  agatedb git:(master) cargo check
    Checking yatp v0.0.1 (https://github.com/tikv/yatp.git#7ed25299)
error[E0599]: no method named `steal_batch_with_limit_and_pop` found for struct `Injector` in the current scope
   --> /root/.cargo/git/checkouts/yatp-e704b73c3ee279b6/7ed2529/src/queue/multilevel.rs:183:37
    |
183 |         self.level_injectors[level].steal_batch_with_limit_and_pop(&self.local_queue, steal_limit)
    |                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: there is a method with a similar name: `steal_batch_and_pop`

For more information about this error, try `rustc --explain E0599`.
error: could not compile `yatp` due to previous error
➜  agatedb git:(master) cd /root/.cargo/git/checkouts/yatp-e704b73c3ee279b6/7ed2529/
➜  7ed2529 git:(master) ✗ cargo check
    Finished dev [unoptimized + debuginfo] target(s) in 0.12s

#192

compact_prios exceeds max_levels

This panic is not guaranteed to be triggered. Will look into details later.

thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Any', /home/skyzh/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool.rs:46:26
stack backtrace:
   0: rust_begin_unwind
             at /rustc/f5230fbf76bafd86ee4376a0e26e551df8d17fec/library/std/src/panicking.rs:495:5
   1: core::panicking::panic_fmt
             at /rustc/f5230fbf76bafd86ee4376a0e26e551df8d17fec/library/core/src/panicking.rs:92:14
   2: core::option::expect_none_failed
             at /rustc/f5230fbf76bafd86ee4376a0e26e551df8d17fec/library/core/src/option.rs:1268:5
   3: core::result::Result<T,E>::unwrap
             at /home/skyzh/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:973:23
   4: yatp::pool::ThreadPool<T>::shutdown
             at /home/skyzh/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool.rs:46:17
   5: <agatedb::db::Agate as core::ops::drop::Drop>::drop
             at ./src/db.rs:98:9
   6: core::ptr::drop_in_place
             at /home/skyzh/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:175:1
   7: agatedb::db::tests::with_agate_test::{{closure}}
             at ./src/db.rs:617:9
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Remove unused functions from binary.rs

#29

Proposal: use ConcurrentSkiplist for memtable

Problem

Now AgateDB will write data in only one thread because the structure skiplist only support one-write-multiple-read. But if there are a large number of keys, it will cost much more time in writing memtable.

Solution

RocksDB has implemented one ConcurrentSkiplist for multiple thread, see more details in https://github.com/facebook/rocksdb/blob/main/memtable/inlineskiplist.h#L907 . I think we can port it into agatedb and then create a threadpool for writing memtable.

[dev] dropPrefixes will cause dead loop

Should iter.next before continue.

https://github.com/tikv/agatedb/blob/develop/src/db.rs#L537

(thanks @soundOfDestiny !)

[dev] panic on drop causes process to abort

As seen in failed CI in #88 , agatedb will abort when we panic on drop. I implemented a partial solution the we ignore all failures (e.g. PoisonedLock, failed to remove file) and simply print them. We could later find a better way to solve this issue.

Feature Reqeust: shard LRU list

AgateDB keep all files open, both SST files and VLog files. If the dataset is large and there are many files, most files may not will touch in a long time. In this scenario we may need a LRU list to maintain those most active files to keep the number of opened files under a specified limit.

Also, after we introduced block cache which used to keep the max memory usage of AgateDB under a limitation to prevent OOM issue, we need the LRU list to maintain those active data blocks.

[dev] lock order inversion in LevelsController

==================
230
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
231
  Cycle in lock order graph: M354 (0x7b1400005000) => M352 (0x7b14000050a0) => M354
232

233
  Mutex M352 acquired here while holding mutex M354 in thread T17:
234
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
235
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
236
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
237
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
238
    #4 agatedb::levels::Core::pick_compact_levels /home/runner/work/agatedb/agatedb/src/levels.rs:659 (agatedb-2d8b3cfdced45fcd+0x2505bb)
239
    #5 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:717 (agatedb-2d8b3cfdced45fcd+0x24174f)
240
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
241
    #7 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
242
    #8 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
243
    #9 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
244
    #10 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
245
    #11 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
246
    #12 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
247
    #13 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
248
    #14 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
249
    #15 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
250
    #16 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
251
    #17 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
252
    #18 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
253
    #19 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
254
    #20 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
255
    #21 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
256
    #22 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
257
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
258
    #24 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
259

260
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
261

262
  Mutex M354 acquired here while holding mutex M352 in thread T17:
263
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
264
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
265
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
266
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
267
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
268
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
269
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
270
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
271
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
272
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
273
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
274
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
275
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
276
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
277
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
278
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
279
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
280
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
281
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
282
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
283
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
284
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
285
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
286
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
287
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
288
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
289
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
290

291
  Thread T17 'agatedb-1' (tid=16933, running) created by thread T15 at:
292
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
293
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
294
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
295
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
296
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
297
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
298
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
299
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
300
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
301
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f3ace)
302
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252f49)
303
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x259779)
304
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122919)
305
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc695a)
306
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
307
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc66c8)
308
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122b79)
309
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x259019)
310
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b797)
311
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
312
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
313
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
314

315
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 in std::sys::unix::rwlock::RWLock::read
316
==================
317
==================
318
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
319
  Cycle in lock order graph: M353 (0x7b1400005230) => M354 (0x7b1400005000) => M353
320

321
  Mutex M354 acquired here while holding mutex M353 in thread T17:
322
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
323
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
324
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
325
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
326
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
327
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
328
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
329
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
330
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
331
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
332
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
333
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
334
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
335
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
336
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
337
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
338
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
339
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
340
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
341
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
342
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
343
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
344
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
345
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
346
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
347
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
348
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
349

350
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
351

352
  Mutex M353 acquired here while holding mutex M354 in thread T17:
353
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
354
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
355
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
356
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
357
    #4 agatedb::levels::compaction::CompactStatus::delete /home/runner/work/agatedb/agatedb/src/levels/compaction.rs:178 (agatedb-2d8b3cfdced45fcd+0xe21db)
358
    #5 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:581 (agatedb-2d8b3cfdced45fcd+0x24f1ad)
359
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
360
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
361
    #8 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
362
    #9 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
363
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
364
    #11 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
365
    #12 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
366
    #13 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
367
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
368
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
369
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
370
    #17 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
371
    #18 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
372
    #19 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
373
    #20 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
374
    #21 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
375
    #22 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
376
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
377
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
378
    #25 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
379

380
  Thread T17 'agatedb-1' (tid=16933, running) created by thread T15 at:
381
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
382
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
383
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
384
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
385
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
386
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
387
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
388
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
389
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
390
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f3ace)
391
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252f49)
392
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x259779)
393
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122919)
394
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc695a)
395
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
396
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc66c8)
397
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122b79)
398
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x259019)
399
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b797)
400
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
401
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
402
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
403

404
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 in std::sys::unix::rwlock::RWLock::write
405
==================
406
==================
407
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
408
  Cycle in lock order graph: M354 (0x7b1400005000) => M353 (0x7b1400005230) => M354
409

410
  Mutex M353 acquired here while holding mutex M354 in thread T17:
411
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
412
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
413
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
414
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
415
    #4 agatedb::levels::compaction::CompactStatus::delete /home/runner/work/agatedb/agatedb/src/levels/compaction.rs:178 (agatedb-2d8b3cfdced45fcd+0xe21db)
416
    #5 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:581 (agatedb-2d8b3cfdced45fcd+0x24f1ad)
417
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
418
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
419
    #8 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
420
    #9 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
421
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
422
    #11 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
423
    #12 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
424
    #13 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
425
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
426
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
427
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
428
    #17 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
429
    #18 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
430
    #19 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
431
    #20 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
432
    #21 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
433
    #22 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
434
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
435
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
436
    #25 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
437

438
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
439

440
  Mutex M354 acquired here while holding mutex M353 in thread T17:
441
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
442
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
443
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
444
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
445
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
446
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
447
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
448
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
449
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
450
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
451
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
452
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
453
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
454
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
455
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
456
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
457
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
458
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
459
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
460
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
461
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
462
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
463
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
464
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
465
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
466
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
467
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
468

469
  Thread T17 'agatedb-1' (tid=16933, running) created by thread T15 at:
470
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
471
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
472
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
473
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
474
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
475
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
476
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
477
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
478
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
479
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f3ace)
480
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252f49)
481
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x259779)
482
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122919)
483
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc695a)
484
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
485
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc66c8)
486
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122b79)
487
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x259019)
488
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b797)
489
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
490
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
491
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
492

493
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 in std::sys::unix::rwlock::RWLock::read
494
==================
495
==================
496
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
497
  Cycle in lock order graph: M352 (0x7b14000050a0) => M354 (0x7b1400005000) => M359 (0x7b1400005370) => M352
498

499
  Mutex M354 acquired here while holding mutex M352 in thread T17:
500
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
501
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
502
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
503
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
504
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
505
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
506
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
507
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
508
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
509
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
510
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
511
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
512
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
513
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
514
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
515
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
516
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
517
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
518
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
519
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
520
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
521
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
522
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
523
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
524
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
525
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
526
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
527

528
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
529

530
  Mutex M359 acquired here while holding mutex M354 in thread T17:
531
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
532
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
533
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
534
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
535
    #4 agatedb::levels::Core::pick_compact_levels /home/runner/work/agatedb/agatedb/src/levels.rs:659 (agatedb-2d8b3cfdced45fcd+0x2505bb)
536
    #5 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:717 (agatedb-2d8b3cfdced45fcd+0x24174f)
537
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
538
    #7 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
539
    #8 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
540
    #9 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
541
    #10 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
542
    #11 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
543
    #12 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
544
    #13 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
545
    #14 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
546
    #15 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
547
    #16 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
548
    #17 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
549
    #18 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
550
    #19 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
551
    #20 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
552
    #21 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
553
    #22 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
554
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
555
    #24 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
556

557
  Mutex M352 acquired here while holding mutex M359 in thread T17:
558
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
559
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
560
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
561
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
562
    #4 agatedb::levels::Core::fill_tables /home/runner/work/agatedb/agatedb/src/levels.rs:233 (agatedb-2d8b3cfdced45fcd+0x24a272)
563
    #5 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:571 (agatedb-2d8b3cfdced45fcd+0x24ed9d)
564
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
565
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
566
    #8 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
567
    #9 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
568
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
569
    #11 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
570
    #12 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
571
    #13 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
572
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
573
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
574
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
575
    #17 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
576
    #18 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
577
    #19 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
578
    #20 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
579
    #21 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
580
    #22 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
581
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
582
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
583
    #25 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
584

585
  Thread T17 'agatedb-1' (tid=16933, running) created by thread T15 at:
586
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
587
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
588
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
589
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
590
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
591
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
592
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
593
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
594
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
595
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f3ace)
596
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252f49)
597
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x259779)
598
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122919)
599
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc695a)
600
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
601
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc66c8)
602
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122b79)
603
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x259019)
604
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b797)
605
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
606
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
607
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
608

609
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 in std::sys::unix::rwlock::RWLock::write
610
==================
611
==================
612
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
613
  Cycle in lock order graph: M344 (0x7b1400002b20) => M342 (0x7b1400002ad0) => M344
614

615
  Mutex M342 acquired here while holding mutex M344 in thread T14:
616
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
617
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
618
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
619
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
620
    #4 agatedb::levels::Core::pick_compact_levels /home/runner/work/agatedb/agatedb/src/levels.rs:659 (agatedb-2d8b3cfdced45fcd+0x2505bb)
621
    #5 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:717 (agatedb-2d8b3cfdced45fcd+0x24174f)
622
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
623
    #7 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
624
    #8 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
625
    #9 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
626
    #10 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
627
    #11 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
628
    #12 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
629
    #13 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
630
    #14 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
631
    #15 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
632
    #16 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
633
    #17 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
634
    #18 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
635
    #19 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
636
    #20 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
637
    #21 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
638
    #22 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
639
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
640
    #24 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
641

642
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
643

644
  Mutex M344 acquired here while holding mutex M342 in thread T14:
645
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
646
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
647
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
648
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
649
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
650
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
651
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
652
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
653
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
654
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
655
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
656
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
657
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
658
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
659
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
660
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
661
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
662
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
663
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
664
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
665
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
666
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
667
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
668
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
669
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
670
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
671
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
672

673
  Thread T14 'agatedb-1' (tid=16930, running) created by thread T11 at:
674
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
675
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
676
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
677
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
678
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
679
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
680
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
681
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
682
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
683
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f404e)
684
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252ea9)
685
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x2597e9)
686
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122989)
687
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc6b3a)
688
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
689
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc6278)
690
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122bf9)
691
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x258bf9)
692
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b6f7)
693
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
694
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
695
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
696

697
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 in std::sys::unix::rwlock::RWLock::read
698
==================
699
==================
700
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
701
  Cycle in lock order graph: M343 (0x7b14000028f0) => M344 (0x7b1400002b20) => M343
702

703
  Mutex M344 acquired here while holding mutex M343 in thread T14:
704
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
705
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
706
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
707
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
708
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
709
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
710
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
711
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
712
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
713
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
714
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
715
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
716
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
717
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
718
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
719
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
720
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
721
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
722
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
723
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
724
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
725
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
726
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
727
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
728
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
729
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
730
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
731

732
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
733

734
  Mutex M343 acquired here while holding mutex M344 in thread T14:
735
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
736
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
737
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
738
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
739
    #4 agatedb::levels::compaction::CompactStatus::delete /home/runner/work/agatedb/agatedb/src/levels/compaction.rs:178 (agatedb-2d8b3cfdced45fcd+0xe21db)
740
    #5 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:581 (agatedb-2d8b3cfdced45fcd+0x24f1ad)
741
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
742
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
743
    #8 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
744
    #9 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
745
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
746
    #11 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
747
    #12 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
748
    #13 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
749
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
750
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
751
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
752
    #17 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
753
    #18 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
754
    #19 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
755
    #20 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
756
    #21 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
757
    #22 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
758
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
759
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
760
    #25 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
761

762
  Thread T14 'agatedb-1' (tid=16930, running) created by thread T11 at:
763
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
764
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
765
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
766
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
767
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
768
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
769
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
770
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
771
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
772
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f404e)
773
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252ea9)
774
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x2597e9)
775
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122989)
776
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc6b3a)
777
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
778
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc6278)
779
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122bf9)
780
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x258bf9)
781
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b6f7)
782
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
783
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
784
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
785

786
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 in std::sys::unix::rwlock::RWLock::write
787
==================
788
==================
789
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
790
  Cycle in lock order graph: M344 (0x7b1400002b20) => M343 (0x7b14000028f0) => M344
791

792
  Mutex M343 acquired here while holding mutex M344 in thread T14:
793
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
794
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
795
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
796
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
797
    #4 agatedb::levels::compaction::CompactStatus::delete /home/runner/work/agatedb/agatedb/src/levels/compaction.rs:178 (agatedb-2d8b3cfdced45fcd+0xe21db)
798
    #5 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:581 (agatedb-2d8b3cfdced45fcd+0x24f1ad)
799
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
800
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
801
    #8 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
802
    #9 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
803
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
804
    #11 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
805
    #12 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
806
    #13 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
807
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
808
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
809
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
810
    #17 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
811
    #18 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
812
    #19 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
813
    #20 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
814
    #21 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
815
    #22 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
816
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
817
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
818
    #25 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
819

820
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
821

822
  Mutex M344 acquired here while holding mutex M343 in thread T14:
823
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
824
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
825
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
826
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
827
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
828
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
829
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
830
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
831
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
832
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
833
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
834
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
835
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
836
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
837
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
838
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
839
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
840
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
841
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
842
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
843
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
844
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
845
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
846
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
847
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
848
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
849
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
850

851
  Thread T14 'agatedb-1' (tid=16930, running) created by thread T11 at:
852
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
853
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
854
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
855
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
856
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
857
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
858
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
859
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
860
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
861
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f404e)
862
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252ea9)
863
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x2597e9)
864
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122989)
865
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc6b3a)
866
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
867
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc6278)
868
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122bf9)
869
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x258bf9)
870
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b6f7)
871
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
872
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
873
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
874

875
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 in std::sys::unix::rwlock::RWLock::read
876
==================
877
==================
878
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
879
  Cycle in lock order graph: M342 (0x7b1400002ad0) => M344 (0x7b1400002b20) => M349 (0x7b1400002a80) => M342
880

881
  Mutex M344 acquired here while holding mutex M342 in thread T14:
882
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
883
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
884
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
885
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
886
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
887
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
888
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
889
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
890
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
891
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
892
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
893
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
894
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
895
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
896
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
897
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
898
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
899
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
900
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
901
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
902
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
903
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
904
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
905
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
906
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
907
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
908
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
909

910
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
911

912
  Mutex M349 acquired here while holding mutex M344 in thread T14:
913
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
914
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
915
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
916
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
917
    #4 agatedb::levels::Core::pick_compact_levels /home/runner/work/agatedb/agatedb/src/levels.rs:659 (agatedb-2d8b3cfdced45fcd+0x2505bb)
918
    #5 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:717 (agatedb-2d8b3cfdced45fcd+0x24174f)
919
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
920
    #7 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
921
    #8 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
922
    #9 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
923
    #10 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
924
    #11 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
925
    #12 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
926
    #13 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
927
    #14 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
928
    #15 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
929
    #16 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
930
    #17 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
931
    #18 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
932
    #19 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
933
    #20 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
934
    #21 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
935
    #22 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
936
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
937
    #24 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
938

939
  Mutex M342 acquired here while holding mutex M349 in thread T14:
940
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
941
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
942
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
943
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
944
    #4 agatedb::levels::Core::fill_tables /home/runner/work/agatedb/agatedb/src/levels.rs:233 (agatedb-2d8b3cfdced45fcd+0x24a272)
945
    #5 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:571 (agatedb-2d8b3cfdced45fcd+0x24ed9d)
946
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
947
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
948
    #8 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
949
    #9 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
950
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
951
    #11 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
952
    #12 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
953
    #13 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
954
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
955
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
956
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
957
    #17 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
958
    #18 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
959
    #19 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
960
    #20 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
961
    #21 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
962
    #22 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
963
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
964
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
965
    #25 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
966

967
  Thread T14 'agatedb-1' (tid=16930, running) created by thread T11 at:
968
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
969
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
970
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
971
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
972
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
973
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
974
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
975
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
976
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
977
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f404e)
978
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252ea9)
979
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x2597e9)
980
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122989)
981
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc6b3a)
982
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
983
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc6278)
984
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122bf9)
985
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x258bf9)
986
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b6f7)
987
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
988
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
989
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
990

991
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 in std::sys::unix::rwlock::RWLock::write
992
==================
993
test db::tests::test_flush_memtable ... ok
994
==================
995
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
996
  Cycle in lock order graph: M344 (0x7b1400002b20) => M343 (0x7b14000028f0) => M344
997

998
  Mutex M343 acquired here while holding mutex M344 in thread T14:
999
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
1000
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
1001
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
1002
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
1003
    #4 agatedb::levels::compaction::CompactStatus::delete /home/runner/work/agatedb/agatedb/src/levels/compaction.rs:178 (agatedb-2d8b3cfdced45fcd+0xe21db)
1004
    #5 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:581 (agatedb-2d8b3cfdced45fcd+0x24f1ad)
1005
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
1006
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
1007
    #8 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
1008
    #9 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
1009
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
1010
    #11 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
1011
    #12 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
1012
    #13 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
1013
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
1014
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
1015
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
1016
    #17 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
1017
    #18 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
1018
    #19 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
1019
    #20 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
1020
    #21 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
1021
    #22 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
1022
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
1023
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
1024
    #25 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
1025

1026
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
1027

1028
  Mutex M344 acquired here while holding mutex M343 in thread T14:
1029
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
1030
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
1031
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
1032
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
1033
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
1034
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
1035
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
1036
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
1037
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
1038
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
1039
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
1040
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
1041
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
1042
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
1043
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
1044
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
1045
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
1046
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
1047
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
1048
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
1049
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
1050
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
1051
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
1052
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
1053
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
1054
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
1055
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
1056

1057
  Thread T14 'agatedb-1' (tid=16930, running) created by thread T11 at:
1058
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
1059
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
1060
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
1061
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
1062
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
1063
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
1064
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
1065
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
1066
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
1067
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f404e)
1068
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252ea9)
1069
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x2597e9)
1070
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122989)
1071
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc6b3a)
1072
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
1073
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc6278)
1074
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122bf9)
1075
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x258bf9)
1076
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b6f7)
1077
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
1078
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
1079
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
1080

1081
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 in std::sys::unix::rwlock::RWLock::read
1082
==================
1083
==================
1084
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16915)
1085
  Cycle in lock order graph: M349 (0x7b1400002a80) => M344 (0x7b1400002b20) => M348 (0x7b1400002a30) => M349
1086

1087
  Mutex M344 acquired here while holding mutex M349 in thread T14:
1088
    #0 pthread_rwlock_wrlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1365 (agatedb-2d8b3cfdced45fcd+0x400f6)
1089
    #1 std::sys::unix::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 (agatedb-2d8b3cfdced45fcd+0x1f016e)
1090
    #2 std::sys_common::rwlock::RWLock::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:48 (agatedb-2d8b3cfdced45fcd+0x1d80d7)
1091
    #3 std::sync::rwlock::RwLock<T>::write /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:264 (agatedb-2d8b3cfdced45fcd+0x198471)
1092
    #4 agatedb::levels::Core::fill_tables_l0_to_lbase /home/runner/work/agatedb/agatedb/src/levels.rs:163 (agatedb-2d8b3cfdced45fcd+0x248cad)
1093
    #5 agatedb::levels::Core::fill_tables_l0 /home/runner/work/agatedb/agatedb/src/levels.rs:224 (agatedb-2d8b3cfdced45fcd+0x249fc7)
1094
    #6 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:558 (agatedb-2d8b3cfdced45fcd+0x24e97c)
1095
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
1096
    #8 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
1097
    #9 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
1098
    #10 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
1099
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
1100
    #12 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
1101
    #13 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
1102
    #14 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
1103
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
1104
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
1105
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
1106
    #18 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
1107
    #19 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
1108
    #20 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
1109
    #21 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
1110
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
1111
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
1112
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
1113
    #25 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
1114
    #26 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
1115

1116
    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
1117

1118
  Mutex M348 acquired here while holding mutex M344 in thread T14:
1119
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
1120
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
1121
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
1122
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
1123
    #4 agatedb::levels::Core::pick_compact_levels /home/runner/work/agatedb/agatedb/src/levels.rs:659 (agatedb-2d8b3cfdced45fcd+0x2505bb)
1124
    #5 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:717 (agatedb-2d8b3cfdced45fcd+0x24174f)
1125
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
1126
    #7 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
1127
    #8 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
1128
    #9 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
1129
    #10 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
1130
    #11 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
1131
    #12 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
1132
    #13 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
1133
    #14 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
1134
    #15 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
1135
    #16 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
1136
    #17 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
1137
    #18 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
1138
    #19 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
1139
    #20 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
1140
    #21 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
1141
    #22 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
1142
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
1143
    #24 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
1144

1145
  Mutex M349 acquired here while holding mutex M348 in thread T14:
1146
    #0 pthread_rwlock_rdlock /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1335 (agatedb-2d8b3cfdced45fcd+0x3fd26)
1147
    #1 std::sys::unix::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:23 (agatedb-2d8b3cfdced45fcd+0x1efdb8)
1148
    #2 std::sys_common::rwlock::RWLock::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/rwlock.rs:26 (agatedb-2d8b3cfdced45fcd+0x1d8077)
1149
    #3 std::sync::rwlock::RwLock<T>::read /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/rwlock.rs:182 (agatedb-2d8b3cfdced45fcd+0x198144)
1150
    #4 agatedb::levels::Core::fill_tables /home/runner/work/agatedb/agatedb/src/levels.rs:233 (agatedb-2d8b3cfdced45fcd+0x24a272)
1151
    #5 agatedb::levels::Core::do_compact /home/runner/work/agatedb/agatedb/src/levels.rs:571 (agatedb-2d8b3cfdced45fcd+0x24ed9d)
1152
    #6 agatedb::levels::LevelsController::run_compactor::{{closure}}::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:732 (agatedb-2d8b3cfdced45fcd+0x241b70)
1153
    #7 agatedb::levels::LevelsController::run_compactor::{{closure}} /home/runner/work/agatedb/agatedb/src/levels.rs:742 (agatedb-2d8b3cfdced45fcd+0x24220f)
1154
    #8 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15c473)
1155
    #9 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b7fb)
1156
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x42986f)
1157
    #11 <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/task/callback.rs:152 (agatedb-2d8b3cfdced45fcd+0x4606b9)
1158
    #12 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:48 (agatedb-2d8b3cfdced45fcd+0x422ca4)
1159
    #13 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-2d8b3cfdced45fcd+0x46763a)
1160
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x42a016)
1161
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x4409aa)
1162
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x43e60a)
1163
    #17 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0x454ad9)
1164
    #18 __rust_try yatp.da9dazvp-cgu.7:? (agatedb-2d8b3cfdced45fcd+0x459c2b)
1165
    #19 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0x454987)
1166
    #20 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x43f6ea)
1167
    #21 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x440727)
1168
    #22 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x41a4cb)
1169
    #23 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
1170
    #24 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
1171
    #25 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
1172

1173
  Thread T14 'agatedb-1' (tid=16930, running) created by thread T11 at:
1174
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-2d8b3cfdced45fcd+0x3da7b)
1175
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-2d8b3cfdced45fcd+0x67582b)
1176
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-2d8b3cfdced45fcd+0x440041)
1177
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-2d8b3cfdced45fcd+0x440a6a)
1178
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-2d8b3cfdced45fcd+0x4671b6)
1179
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-2d8b3cfdced45fcd+0x467d91)
1180
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-2d8b3cfdced45fcd+0x467c8c)
1181
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:72 (agatedb-2d8b3cfdced45fcd+0x17e172)
1182
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:233 (agatedb-2d8b3cfdced45fcd+0x137bd4)
1183
    #9 agatedb::db::tests::with_agate_test::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:618 (agatedb-2d8b3cfdced45fcd+0x1f404e)
1184
    #10 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-2d8b3cfdced45fcd+0x252ea9)
1185
    #11 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-2d8b3cfdced45fcd+0x2597e9)
1186
    #12 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-2d8b3cfdced45fcd+0x122989)
1187
    #13 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379 (agatedb-2d8b3cfdced45fcd+0xc6b3a)
1188
    #14 __rust_try 1dzu5xa3oeqaq90p:? (agatedb-2d8b3cfdced45fcd+0xc744b)
1189
    #15 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343 (agatedb-2d8b3cfdced45fcd+0xc6278)
1190
    #16 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-2d8b3cfdced45fcd+0x122bf9)
1191
    #17 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-2d8b3cfdced45fcd+0x258bf9)
1192
    #18 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-2d8b3cfdced45fcd+0x15b6f7)
1193
    #19 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x630f3b)
1194
    #20 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1318 (agatedb-2d8b3cfdced45fcd+0x631036)
1195
    #21 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-2d8b3cfdced45fcd+0x675cb3)
1196

1197
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/rwlock.rs:73 in std::sys::unix::rwlock::RWLock::write

Segmentation violation on leak sanitizer test

As shown in #143 (comment), we may not pass the leak sanitizer test due to the SEGV error:

Stack trace

test ops::transaction_test::normal_db::test_conflict ... LeakSanitizer:DEADLYSIGNAL
==9609==ERROR: LeakSanitizer: SEGV on unknown address 0x7f7858d75ff8 (pc 0x55cbb28f0094 bp 0x7f780e4c68e0 sp 0x7f780e4c6060 T1023)
==9609==The signal is caused by a READ memory access.
    #0 0x55cbb28f0094  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x86094) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #1 0x55cbb28ea3e2  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x803e2) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #2 0x55cbb28eea2f  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x84a2f) (BuildId: 0[219](https://github.com/tikv/agatedb/runs/6933910360?check_suite_focus=true#step:5:220)e8618c4e0fa0937fde79c12d993f79cbee5f)
    #3 0x55cbb2d502ae  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x4e62ae) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #4 0x55cbb2dd92b1  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x56f2b1) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #5 0x55cbb2ea2709  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x638709) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #6 0x55cbb2ea2b44  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x638b44) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #7 0x55cbb2ea2f1f  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x638f1f) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #8 0x55cbb2e9bcfb  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x631cfb) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #9 0x55cbb2e9b396  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x631396) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #10 0x55cbb2e9aef8  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x630ef8) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #11 0x55cbb2e9aedd  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x630edd) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #12 0x55cbb2e9fc50  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x635c50) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #13 0x55cbb2d22e12  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x4b8e12) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #14 0x55cbb29[228](https://github.com/tikv/agatedb/runs/6933910360?check_suite_focus=true#step:5:229)00  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0xb8800) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #15 0x55cbb2936721  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0xcc721) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #16 0x55cbb2a2772e  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x1bd72e) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #17 0x55cbb2a9b721  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x[231](https://github.com/tikv/agatedb/runs/6933910360?check_suite_focus=true#step:5:232)721) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #18 0x55cbb2ae4714  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x27a714) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #19 0x55cbb299b857  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x131857) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #20 0x55cbb2ae56f7  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x27b6f7) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #21 0x55cbb2a750ef  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x20b0ef) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #22 0x55cbb295821b  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0xee21b) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #23 0x55cbb29fd75f  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x19375f) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #24 0x55cbb2924926  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0xba926) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #25 0x55cbb2925a2a  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0xbba2a) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #26 0x55cbb2923ed0  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0xb9ed0) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #27 0x55cbb296850f  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0xfe50f) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #28 0x55cbb2953fa2  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0xe9fa2) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #29 0x55cbb299a5dd  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x1305dd) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #30 0x55cbb2e4b32c  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x5e132c) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #31 0x55cbb2e4b39f  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x5e139f) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #32 0x55cbb2d52501  (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x4e8501) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f)
    #33 0x7f7859879608  (/lib/x86_64-linux-gnu/libpthread.so.0+0x8608) (BuildId: 7b4536f41cdaa5888408e82d0836e33dcf436466)
    #34 0x7f7859649132  (/lib/x86_64-linux-gnu/libc.so.6+0x11f132) (BuildId: 1878e6b475720c7c51969e69ab2d276fae6d1dee)
LeakSanitizer can not provide additional info.
SUMMARY: LeakSanitizer: SEGV (/home/runner/work/agatedb/agatedb/target/x86_64-unknown-linux-gnu/debug/deps/agatedb-da9f22d8c6de7660+0x86094) (BuildId: 0219e8618c4e0fa0937fde79c12d993f79cbee5f) 
==9609==ABORTING
error: test failed, to rerun pass '--lib'
Error: The process '/home/runner/.cargo/bin/cargo' failed with exit code 23

If we test it locally, we could additionally get error information as below.

Additional error information

error: address range table at offset 0x58280 has a premature terminator entry at offset 0x582b0
error: address range table at offset 0x58280 has a premature terminator entry at offset 0x582d0
error: address range table at offset 0x58280 has a premature terminator entry at offset 0x582f0
error: address range table at offset 0x687a0 has a premature terminator entry at offset 0x68860
error: address range table at offset 0x687a0 has a premature terminator entry at offset 0x68880
error: address range table at offset 0x72700 has a premature terminator entry at offset 0x72780
error: address range table at offset 0x72700 has a premature terminator entry at offset 0x727a0
error: address range table at offset 0x73510 has a premature terminator entry at offset 0x73540
error: address range table at offset 0x82280 has a premature terminator entry at offset 0x82290

Current workaround is running tests serially and reduce the concurrency in a certain test.

Not sure the cause of this issue.

[dev] pending_write in txn cannot be read

https://github.com/tikv/agatedb/blob/develop/src/ops/transaction.rs#L228

should return Ok when pending_write has been found.

(thanks @soundOfDestiny)!

Add sanitizer check

There are some unsafe code in the project, enabling sanitizer check can detect memory issues in the early stage and provide more confidence in memory safety.

This is a guide about how to enable the check: https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/sanitizer.html.

能给一些入门使用的例子吗

data race in thread sanitizer test

#49

WARNING: ThreadSanitizer: data race (pid=17810)
191
  Write of size 8 at 0x7b740000fa00 by thread T11:
192
    #0 free /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:708 (agatedb-6848b01562dc6aa7+0x39d38)
193
    #1 std::sys::unix::alloc::<impl core::alloc::global::GlobalAlloc for std::alloc::System>::dealloc /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/alloc.rs:42 (agatedb-6848b01562dc6aa7+0x5814a5)
194
    #2 __rdl_dealloc /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/alloc.rs:364 (agatedb-6848b01562dc6aa7+0x4e6040)
195
    #3 alloc::alloc::dealloc /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:102 (agatedb-6848b01562dc6aa7+0x40c368)
196
    #4 <alloc::alloc::Global as core::alloc::AllocRef>::dealloc /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:237 (agatedb-6848b01562dc6aa7+0x40cd57)
197
    #5 alloc::alloc::box_free /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:332 (agatedb-6848b01562dc6aa7+0x40c629)
198
    #6 core::ptr::drop_in_place /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:175 (agatedb-6848b01562dc6aa7+0x3fe5da)
199
    #7 core::mem::drop /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/mem/mod.rs:882 (agatedb-6848b01562dc6aa7+0x3f999f)
200
    #8 <crossbeam_epoch::atomic::Owned<T> as core::ops::drop::Drop>::drop /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/atomic.rs:760 (agatedb-6848b01562dc6aa7+0x405c1c)
201
    #9 core::ptr::drop_in_place /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:175 (agatedb-6848b01562dc6aa7+0x3fe2bb)
202
    #10 core::mem::drop /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/mem/mod.rs:882 (agatedb-6848b01562dc6aa7+0x3f97ff)
203
    #11 crossbeam_epoch::guard::Guard::defer_unchecked::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/guard.rs:195 (agatedb-6848b01562dc6aa7+0x40b72e)
204
    #12 crossbeam_epoch::deferred::Deferred::new::call /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/deferred.rs:46 (agatedb-6848b01562dc6aa7+0x4089a6)
205
    #13 crossbeam_epoch::deferred::Deferred::call /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/deferred.rs:77 (agatedb-6848b01562dc6aa7+0x408c95)
206
    #14 <crossbeam_epoch::internal::Bag as core::ops::drop::Drop>::drop /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:201 (agatedb-6848b01562dc6aa7+0x3f6cef)
207
    #15 core::ptr::drop_in_place /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:175 (agatedb-6848b01562dc6aa7+0x3fdf5b)
208
    #16 core::ptr::drop_in_place /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:175 (agatedb-6848b01562dc6aa7+0x3fe00f)
209
    #17 core::mem::drop /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/mem/mod.rs:882 (agatedb-6848b01562dc6aa7+0x3f9853)
210
    #18 crossbeam_epoch::internal::Global::collect /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:295 (agatedb-6848b01562dc6aa7+0x3f7410)
211
    #19 crossbeam_epoch::internal::Local::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:497 (agatedb-6848b01562dc6aa7+0x382b77)
212
    #20 crossbeam_epoch::collector::LocalHandle::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/collector.rs:75 (agatedb-6848b01562dc6aa7+0x3b3474)
213
    #21 crossbeam_epoch::default::pin::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:23 (agatedb-6848b01562dc6aa7+0x381eb6)
214
    #22 crossbeam_epoch::default::with_handle::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:43 (agatedb-6848b01562dc6aa7+0x381b54)
215
    #23 std::thread::local::LocalKey<T>::try_with /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:272 (agatedb-6848b01562dc6aa7+0x398688)
216
    #24 crossbeam_epoch::default::with_handle /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:42 (agatedb-6848b01562dc6aa7+0x381a9c)
217
    #25 crossbeam_epoch::default::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:23 (agatedb-6848b01562dc6aa7+0x381e46)
218
    #26 crossbeam_deque::Stealer<T>::steal_batch_and_pop /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-deque-0.7.3/src/lib.rs:956 (agatedb-6848b01562dc6aa7+0x3a9b19)
219
    #27 yatp::queue::single_level::LocalQueue<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/queue/single_level.rs:87 (agatedb-6848b01562dc6aa7+0x3cefd4)
220
    #28 yatp::queue::LocalQueue<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/queue.rs:98 (agatedb-6848b01562dc6aa7+0x39bfee)
221
    #29 yatp::pool::spawn::Local<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/spawn.rs:255 (agatedb-6848b01562dc6aa7+0x3ce4e1)
222
    #30 yatp::pool::worker::WorkerThread<T,R>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:28 (agatedb-6848b01562dc6aa7+0x39b926)
223
    #31 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:44 (agatedb-6848b01562dc6aa7+0x39bbff)
224
    #32 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-6848b01562dc6aa7+0x3d44da)
225
    #33 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-6848b01562dc6aa7+0x398126)
226
    #34 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-6848b01562dc6aa7+0x3b49da)
227
    #35 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x3a049a)
228
    #36 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x3d1609)
229
    #37 __rust_try yatp.exejrn79-cgu.8:? (agatedb-6848b01562dc6aa7+0x3d584b)
230
    #38 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x3d14b7)
231
    #39 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x3a0b6a)
232
    #40 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-6848b01562dc6aa7+0x3b4723)
233
    #41 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x38342b)
234
    #42 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557b0b)
235
    #43 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557c06)
236
    #44 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-6848b01562dc6aa7+0x519e03)
237

238
  Previous atomic read of size 8 at 0x7b740000fa00 by thread T7:
239
    #0 __tsan_atomic64_load /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp:539 (agatedb-6848b01562dc6aa7+0x7cf1e)
240
    #1 core::sync::atomic::atomic_load /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/sync/atomic.rs:2354 (agatedb-6848b01562dc6aa7+0x710a8e)
241
    #2 core::sync::atomic::AtomicUsize::load /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/sync/atomic.rs:1486 (agatedb-6848b01562dc6aa7+0x3fca06)
242
    #3 crossbeam_epoch::atomic::Atomic<T>::load /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/atomic.rs:209 (agatedb-6848b01562dc6aa7+0x404886)
243
    #4 <crossbeam_epoch::sync::list::Iter<T,C> as core::iter::traits::iterator::Iterator>::next /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/sync/list.rs:240 (agatedb-6848b01562dc6aa7+0x40716e)
244
    #5 crossbeam_epoch::internal::Global::try_advance /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:316 (agatedb-6848b01562dc6aa7+0x3f760c)
245
    #6 crossbeam_epoch::internal::Global::collect /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:281 (agatedb-6848b01562dc6aa7+0x3f7265)
246
    #7 crossbeam_epoch::internal::Local::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:497 (agatedb-6848b01562dc6aa7+0x382b77)
247
    #8 crossbeam_epoch::collector::LocalHandle::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/collector.rs:75 (agatedb-6848b01562dc6aa7+0x3b3474)
248
    #9 crossbeam_epoch::default::pin::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:23 (agatedb-6848b01562dc6aa7+0x381eb6)
249
    #10 crossbeam_epoch::default::with_handle::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:43 (agatedb-6848b01562dc6aa7+0x381b54)
250
    #11 std::thread::local::LocalKey<T>::try_with /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:272 (agatedb-6848b01562dc6aa7+0x398688)
251
    #12 crossbeam_epoch::default::with_handle /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:42 (agatedb-6848b01562dc6aa7+0x381a9c)
252
    #13 crossbeam_epoch::default::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:23 (agatedb-6848b01562dc6aa7+0x381e46)
253
    #14 crossbeam_deque::Stealer<T>::steal_batch_and_pop /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-deque-0.7.3/src/lib.rs:956 (agatedb-6848b01562dc6aa7+0x3a9b19)
254
    #15 yatp::queue::single_level::LocalQueue<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/queue/single_level.rs:87 (agatedb-6848b01562dc6aa7+0x3cefd4)
255
    #16 yatp::queue::LocalQueue<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/queue.rs:98 (agatedb-6848b01562dc6aa7+0x39bfee)
256
    #17 yatp::pool::spawn::Local<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/spawn.rs:255 (agatedb-6848b01562dc6aa7+0x3ce4e1)
257
    #18 yatp::pool::worker::WorkerThread<T,R>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:28 (agatedb-6848b01562dc6aa7+0x39b926)
258
    #19 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:44 (agatedb-6848b01562dc6aa7+0x39bbff)
259
    #20 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-6848b01562dc6aa7+0x3d44da)
260
    #21 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-6848b01562dc6aa7+0x398126)
261
    #22 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-6848b01562dc6aa7+0x3b49da)
262
    #23 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x3a049a)
263
    #24 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x3d1609)
264
    #25 __rust_try yatp.exejrn79-cgu.8:? (agatedb-6848b01562dc6aa7+0x3d584b)
265
    #26 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x3d14b7)
266
    #27 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x3a0b6a)
267
    #28 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-6848b01562dc6aa7+0x3b4723)
268
    #29 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x38342b)
269
    #30 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557b0b)
270
    #31 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557c06)
271
    #32 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-6848b01562dc6aa7+0x519e03)
272

273
  Thread T11 'agatedb-1' (tid=17822, running) created by thread T9 at:
274
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-6848b01562dc6aa7+0x3ac9b)
275
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-6848b01562dc6aa7+0x51997b)
276
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-6848b01562dc6aa7+0x3b3eef)
277
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-6848b01562dc6aa7+0x3b4a9a)
278
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-6848b01562dc6aa7+0x3d4056)
279
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-6848b01562dc6aa7+0x3d4c31)
280
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-6848b01562dc6aa7+0x3d4b2c)
281
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:69 (agatedb-6848b01562dc6aa7+0xae5d3)
282
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:216 (agatedb-6848b01562dc6aa7+0x13ef74)
283
    #9 agatedb::db::tests::with_agate_test /home/runner/work/agatedb/agatedb/src/db.rs:577 (agatedb-6848b01562dc6aa7+0xb8c10)
284
    #10 agatedb::db::tests::test_flush_memtable_bigvalue /home/runner/work/agatedb/agatedb/src/db.rs:660 (agatedb-6848b01562dc6aa7+0xf37f2)
285
    #11 agatedb::db::tests::test_flush_memtable_bigvalue::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:659 (agatedb-6848b01562dc6aa7+0xba864)
286
    #12 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x18eb87)
287
    #13 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a43a1)
288
    #14 test::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:517 (agatedb-6848b01562dc6aa7+0x259afb)
289
    #15 test::run_test::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:508 (agatedb-6848b01562dc6aa7+0x2590d7)
290
    #16 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a4114)
291
    #17 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x2335bb)
292
    #18 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x2a0ddd)
293
    #19 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x203ba2)
294
    #20 __rust_try test.cvxxcx5e-cgu.1:? (agatedb-6848b01562dc6aa7+0x20deab)
295
    #21 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x2039b8)
296
    #22 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x2a31bd)
297
    #23 test::run_test_in_process /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:544 (agatedb-6848b01562dc6aa7+0x25a02d)
298
    #24 test::run_test::run_test_inner::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:450 (agatedb-6848b01562dc6aa7+0x25989f)
299
    #25 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-6848b01562dc6aa7+0x2a27b6)
300
    #26 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-6848b01562dc6aa7+0x22cc8a)
301
    #27 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x2a0d6a)
302
    #28 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x203af9)
303
    #29 __rust_try test.cvxxcx5e-cgu.1:? (agatedb-6848b01562dc6aa7+0x20deab)
304
    #30 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x203357)
305
    #31 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x2a324a)
306
    #32 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-6848b01562dc6aa7+0x22c9d3)
307
    #33 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a41eb)
308
    #34 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557b0b)
309
    #35 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557c06)
310
    #36 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-6848b01562dc6aa7+0x519e03)
311

312
  Thread T7 'agatedb-1' (tid=17818, running) created by thread T4 at:
313
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-6848b01562dc6aa7+0x3ac9b)
314
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-6848b01562dc6aa7+0x51997b)
315
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-6848b01562dc6aa7+0x3b3eef)
316
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-6848b01562dc6aa7+0x3b4a9a)
317
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-6848b01562dc6aa7+0x3d4056)
318
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-6848b01562dc6aa7+0x3d4c31)
319
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-6848b01562dc6aa7+0x3d4b2c)
320
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:69 (agatedb-6848b01562dc6aa7+0xae5d3)
321
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:216 (agatedb-6848b01562dc6aa7+0x13ef74)
322
    #9 agatedb::db::tests::with_agate_test /home/runner/work/agatedb/agatedb/src/db.rs:577 (agatedb-6848b01562dc6aa7+0xb9330)
323
    #10 agatedb::db::tests::test_flush_memtable /home/runner/work/agatedb/agatedb/src/db.rs:652 (agatedb-6848b01562dc6aa7+0xf37b2)
324
    #11 agatedb::db::tests::test_flush_memtable::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:651 (agatedb-6848b01562dc6aa7+0xba724)
325
    #12 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x18e917)
326
    #13 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a43a1)
327
    #14 test::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:517 (agatedb-6848b01562dc6aa7+0x259afb)
328
    #15 test::run_test::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:508 (agatedb-6848b01562dc6aa7+0x2590d7)
329
    #16 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a4114)
330
    #17 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x2335bb)
331
    #18 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x2a0ddd)
332
    #19 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x203ba2)
333
    #20 __rust_try test.cvxxcx5e-cgu.1:? (agatedb-6848b01562dc6aa7+0x20deab)
334
    #21 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x2039b8)
335
    #22 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x2a31bd)
336
    #23 test::run_test_in_process /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:544 (agatedb-6848b01562dc6aa7+0x25a02d)
337
    #24 test::run_test::run_test_inner::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:450 (agatedb-6848b01562dc6aa7+0x25989f)
338
    #25 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-6848b01562dc6aa7+0x2a27b6)
339
    #26 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-6848b01562dc6aa7+0x22cc8a)
340
    #27 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x2a0d6a)
341
    #28 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x203af9)
342
    #29 __rust_try test.cvxxcx5e-cgu.1:? (agatedb-6848b01562dc6aa7+0x20deab)
343
    #30 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x203357)
344
    #31 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x2a324a)
345
    #32 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-6848b01562dc6aa7+0x22c9d3)
346
    #33 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a41eb)
347
    #34 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557b0b)
348
    #35 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557c06)
349
    #36 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-6848b01562dc6aa7+0x519e03)
350

351
SUMMARY: ThreadSanitizer: data race /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/alloc.rs:42 in std::sys::unix::alloc::<impl core::alloc::global::GlobalAlloc for std::alloc::System>::dealloc
352
==================
353
==================
354
WARNING: ThreadSanitizer: data race (pid=17810)
355
  Write of size 8 at 0x7b740000fa08 by thread T11:
356
    #0 free /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:708 (agatedb-6848b01562dc6aa7+0x39d38)
357
    #1 std::sys::unix::alloc::<impl core::alloc::global::GlobalAlloc for std::alloc::System>::dealloc /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/alloc.rs:42 (agatedb-6848b01562dc6aa7+0x5814a5)
358
    #2 __rdl_dealloc /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/alloc.rs:364 (agatedb-6848b01562dc6aa7+0x4e6040)
359
    #3 alloc::alloc::dealloc /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:102 (agatedb-6848b01562dc6aa7+0x40c368)
360
    #4 <alloc::alloc::Global as core::alloc::AllocRef>::dealloc /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:237 (agatedb-6848b01562dc6aa7+0x40cd57)
361
    #5 alloc::alloc::box_free /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/alloc.rs:332 (agatedb-6848b01562dc6aa7+0x40c629)
362
    #6 core::ptr::drop_in_place /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:175 (agatedb-6848b01562dc6aa7+0x3fe5da)
363
    #7 core::mem::drop /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/mem/mod.rs:882 (agatedb-6848b01562dc6aa7+0x3f999f)
364
    #8 <crossbeam_epoch::atomic::Owned<T> as core::ops::drop::Drop>::drop /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/atomic.rs:760 (agatedb-6848b01562dc6aa7+0x405c1c)
365
    #9 core::ptr::drop_in_place /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:175 (agatedb-6848b01562dc6aa7+0x3fe2bb)
366
    #10 core::mem::drop /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/mem/mod.rs:882 (agatedb-6848b01562dc6aa7+0x3f97ff)
367
    #11 crossbeam_epoch::guard::Guard::defer_unchecked::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/guard.rs:195 (agatedb-6848b01562dc6aa7+0x40b72e)
368
    #12 crossbeam_epoch::deferred::Deferred::new::call /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/deferred.rs:46 (agatedb-6848b01562dc6aa7+0x4089a6)
369
    #13 crossbeam_epoch::deferred::Deferred::call /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/deferred.rs:77 (agatedb-6848b01562dc6aa7+0x408c95)
370
    #14 <crossbeam_epoch::internal::Bag as core::ops::drop::Drop>::drop /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:201 (agatedb-6848b01562dc6aa7+0x3f6cef)
371
    #15 core::ptr::drop_in_place /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:175 (agatedb-6848b01562dc6aa7+0x3fdf5b)
372
    #16 core::ptr::drop_in_place /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:175 (agatedb-6848b01562dc6aa7+0x3fe00f)
373
    #17 core::mem::drop /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/mem/mod.rs:882 (agatedb-6848b01562dc6aa7+0x3f9853)
374
    #18 crossbeam_epoch::internal::Global::collect /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:295 (agatedb-6848b01562dc6aa7+0x3f7410)
375
    #19 crossbeam_epoch::internal::Local::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:497 (agatedb-6848b01562dc6aa7+0x382b77)
376
    #20 crossbeam_epoch::collector::LocalHandle::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/collector.rs:75 (agatedb-6848b01562dc6aa7+0x3b3474)
377
    #21 crossbeam_epoch::default::pin::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:23 (agatedb-6848b01562dc6aa7+0x381eb6)
378
    #22 crossbeam_epoch::default::with_handle::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:43 (agatedb-6848b01562dc6aa7+0x381b54)
379
    #23 std::thread::local::LocalKey<T>::try_with /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:272 (agatedb-6848b01562dc6aa7+0x398688)
380
    #24 crossbeam_epoch::default::with_handle /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:42 (agatedb-6848b01562dc6aa7+0x381a9c)
381
    #25 crossbeam_epoch::default::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:23 (agatedb-6848b01562dc6aa7+0x381e46)
382
    #26 crossbeam_deque::Stealer<T>::steal_batch_and_pop /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-deque-0.7.3/src/lib.rs:956 (agatedb-6848b01562dc6aa7+0x3a9b19)
383
    #27 yatp::queue::single_level::LocalQueue<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/queue/single_level.rs:87 (agatedb-6848b01562dc6aa7+0x3cefd4)
384
    #28 yatp::queue::LocalQueue<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/queue.rs:98 (agatedb-6848b01562dc6aa7+0x39bfee)
385
    #29 yatp::pool::spawn::Local<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/spawn.rs:255 (agatedb-6848b01562dc6aa7+0x3ce4e1)
386
    #30 yatp::pool::worker::WorkerThread<T,R>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:28 (agatedb-6848b01562dc6aa7+0x39b926)
387
    #31 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:44 (agatedb-6848b01562dc6aa7+0x39bbff)
388
    #32 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-6848b01562dc6aa7+0x3d44da)
389
    #33 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-6848b01562dc6aa7+0x398126)
390
    #34 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-6848b01562dc6aa7+0x3b49da)
391
    #35 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x3a049a)
392
    #36 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x3d1609)
393
    #37 __rust_try yatp.exejrn79-cgu.8:? (agatedb-6848b01562dc6aa7+0x3d584b)
394
    #38 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x3d14b7)
395
    #39 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x3a0b6a)
396
    #40 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-6848b01562dc6aa7+0x3b4723)
397
    #41 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x38342b)
398
    #42 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557b0b)
399
    #43 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557c06)
400
    #44 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-6848b01562dc6aa7+0x519e03)
401

402
  Previous atomic read of size 8 at 0x7b740000fa08 by thread T7:
403
    #0 __tsan_atomic64_load /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp:539 (agatedb-6848b01562dc6aa7+0x7cf1e)
404
    #1 core::sync::atomic::atomic_load /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/sync/atomic.rs:2355 (agatedb-6848b01562dc6aa7+0x710aa9)
405
    #2 core::sync::atomic::AtomicUsize::load /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/sync/atomic.rs:1486 (agatedb-6848b01562dc6aa7+0x3fca06)
406
    #3 crossbeam_epoch::epoch::AtomicEpoch::load /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/epoch.rs:93 (agatedb-6848b01562dc6aa7+0x40b2d7)
407
    #4 crossbeam_epoch::internal::Global::try_advance /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:325 (agatedb-6848b01562dc6aa7+0x3f76c6)
408
    #5 crossbeam_epoch::internal::Global::collect /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:281 (agatedb-6848b01562dc6aa7+0x3f7265)
409
    #6 crossbeam_epoch::internal::Local::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/internal.rs:497 (agatedb-6848b01562dc6aa7+0x382b77)
410
    #7 crossbeam_epoch::collector::LocalHandle::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/collector.rs:75 (agatedb-6848b01562dc6aa7+0x3b3474)
411
    #8 crossbeam_epoch::default::pin::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:23 (agatedb-6848b01562dc6aa7+0x381eb6)
412
    #9 crossbeam_epoch::default::with_handle::{{closure}} /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:43 (agatedb-6848b01562dc6aa7+0x381b54)
413
    #10 std::thread::local::LocalKey<T>::try_with /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:272 (agatedb-6848b01562dc6aa7+0x398688)
414
    #11 crossbeam_epoch::default::with_handle /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:42 (agatedb-6848b01562dc6aa7+0x381a9c)
415
    #12 crossbeam_epoch::default::pin /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/default.rs:23 (agatedb-6848b01562dc6aa7+0x381e46)
416
    #13 crossbeam_deque::Stealer<T>::steal_batch_and_pop /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-deque-0.7.3/src/lib.rs:956 (agatedb-6848b01562dc6aa7+0x3a9b19)
417
    #14 yatp::queue::single_level::LocalQueue<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/queue/single_level.rs:87 (agatedb-6848b01562dc6aa7+0x3cefd4)
418
    #15 yatp::queue::LocalQueue<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/queue.rs:98 (agatedb-6848b01562dc6aa7+0x39bfee)
419
    #16 yatp::pool::spawn::Local<T>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/spawn.rs:255 (agatedb-6848b01562dc6aa7+0x3ce4e1)
420
    #17 yatp::pool::worker::WorkerThread<T,R>::pop /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:28 (agatedb-6848b01562dc6aa7+0x39b926)
421
    #18 yatp::pool::worker::WorkerThread<T,R>::run /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/worker.rs:44 (agatedb-6848b01562dc6aa7+0x39bbff)
422
    #19 yatp::pool::builder::LazyBuilder<T>::build::{{closure}} /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:91 (agatedb-6848b01562dc6aa7+0x3d44da)
423
    #20 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-6848b01562dc6aa7+0x398126)
424
    #21 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-6848b01562dc6aa7+0x3b49da)
425
    #22 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x3a049a)
426
    #23 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x3d1609)
427
    #24 __rust_try yatp.exejrn79-cgu.8:? (agatedb-6848b01562dc6aa7+0x3d584b)
428
    #25 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x3d14b7)
429
    #26 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x3a0b6a)
430
    #27 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-6848b01562dc6aa7+0x3b4723)
431
    #28 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x38342b)
432
    #29 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557b0b)
433
    #30 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557c06)
434
    #31 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-6848b01562dc6aa7+0x519e03)
435

436
  Thread T11 'agatedb-1' (tid=17822, running) created by thread T9 at:
437
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-6848b01562dc6aa7+0x3ac9b)
438
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-6848b01562dc6aa7+0x51997b)
439
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-6848b01562dc6aa7+0x3b3eef)
440
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-6848b01562dc6aa7+0x3b4a9a)
441
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-6848b01562dc6aa7+0x3d4056)
442
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-6848b01562dc6aa7+0x3d4c31)
443
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-6848b01562dc6aa7+0x3d4b2c)
444
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:69 (agatedb-6848b01562dc6aa7+0xae5d3)
445
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:216 (agatedb-6848b01562dc6aa7+0x13ef74)
446
    #9 agatedb::db::tests::with_agate_test /home/runner/work/agatedb/agatedb/src/db.rs:577 (agatedb-6848b01562dc6aa7+0xb8c10)
447
    #10 agatedb::db::tests::test_flush_memtable_bigvalue /home/runner/work/agatedb/agatedb/src/db.rs:660 (agatedb-6848b01562dc6aa7+0xf37f2)
448
    #11 agatedb::db::tests::test_flush_memtable_bigvalue::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:659 (agatedb-6848b01562dc6aa7+0xba864)
449
    #12 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x18eb87)
450
    #13 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a43a1)
451
    #14 test::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:517 (agatedb-6848b01562dc6aa7+0x259afb)
452
    #15 test::run_test::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:508 (agatedb-6848b01562dc6aa7+0x2590d7)
453
    #16 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a4114)
454
    #17 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x2335bb)
455
    #18 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x2a0ddd)
456
    #19 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x203ba2)
457
    #20 __rust_try test.cvxxcx5e-cgu.1:? (agatedb-6848b01562dc6aa7+0x20deab)
458
    #21 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x2039b8)
459
    #22 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x2a31bd)
460
    #23 test::run_test_in_process /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:544 (agatedb-6848b01562dc6aa7+0x25a02d)
461
    #24 test::run_test::run_test_inner::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:450 (agatedb-6848b01562dc6aa7+0x25989f)
462
    #25 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-6848b01562dc6aa7+0x2a27b6)
463
    #26 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-6848b01562dc6aa7+0x22cc8a)
464
    #27 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x2a0d6a)
465
    #28 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x203af9)
466
    #29 __rust_try test.cvxxcx5e-cgu.1:? (agatedb-6848b01562dc6aa7+0x20deab)
467
    #30 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x203357)
468
    #31 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x2a324a)
469
    #32 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-6848b01562dc6aa7+0x22c9d3)
470
    #33 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a41eb)
471
    #34 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557b0b)
472
    #35 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557c06)
473
    #36 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-6848b01562dc6aa7+0x519e03)
474

475
  Thread T7 'agatedb-1' (tid=17818, running) created by thread T4 at:
476
    #0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (agatedb-6848b01562dc6aa7+0x3ac9b)
477
    #1 std::sys::unix::thread::Thread::new /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:50 (agatedb-6848b01562dc6aa7+0x51997b)
478
    #2 std::thread::Builder::spawn_unchecked /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:498 (agatedb-6848b01562dc6aa7+0x3b3eef)
479
    #3 std::thread::Builder::spawn /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:381 (agatedb-6848b01562dc6aa7+0x3b4a9a)
480
    #4 yatp::pool::builder::LazyBuilder<T>::build /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:89 (agatedb-6848b01562dc6aa7+0x3d4056)
481
    #5 yatp::pool::builder::Builder::build_with_queue_and_runner /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:260 (agatedb-6848b01562dc6aa7+0x3d4c31)
482
    #6 yatp::pool::builder::Builder::build_callback_pool /home/runner/.cargo/git/checkouts/yatp-e704b73c3ee279b6/6bbea16/src/pool/builder.rs:225 (agatedb-6848b01562dc6aa7+0x3d4b2c)
483
    #7 agatedb::db::Agate::new /home/runner/work/agatedb/agatedb/src/db.rs:69 (agatedb-6848b01562dc6aa7+0xae5d3)
484
    #8 agatedb::db::AgateOptions::open /home/runner/work/agatedb/agatedb/src/db.rs:216 (agatedb-6848b01562dc6aa7+0x13ef74)
485
    #9 agatedb::db::tests::with_agate_test /home/runner/work/agatedb/agatedb/src/db.rs:577 (agatedb-6848b01562dc6aa7+0xb9330)
486
    #10 agatedb::db::tests::test_flush_memtable /home/runner/work/agatedb/agatedb/src/db.rs:652 (agatedb-6848b01562dc6aa7+0xf37b2)
487
    #11 agatedb::db::tests::test_flush_memtable::{{closure}} /home/runner/work/agatedb/agatedb/src/db.rs:651 (agatedb-6848b01562dc6aa7+0xba724)
488
    #12 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x18e917)
489
    #13 core::ops::function::FnOnce::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a43a1)
490
    #14 test::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:517 (agatedb-6848b01562dc6aa7+0x259afb)
491
    #15 test::run_test::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:508 (agatedb-6848b01562dc6aa7+0x2590d7)
492
    #16 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a4114)
493
    #17 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x2335bb)
494
    #18 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x2a0ddd)
495
    #19 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x203ba2)
496
    #20 __rust_try test.cvxxcx5e-cgu.1:? (agatedb-6848b01562dc6aa7+0x20deab)
497
    #21 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x2039b8)
498
    #22 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x2a31bd)
499
    #23 test::run_test_in_process /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:544 (agatedb-6848b01562dc6aa7+0x25a02d)
500
    #24 test::run_test::run_test_inner::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:450 (agatedb-6848b01562dc6aa7+0x25989f)
501
    #25 std::sys_common::backtrace::__rust_begin_short_backtrace /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125 (agatedb-6848b01562dc6aa7+0x2a27b6)
502
    #26 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474 (agatedb-6848b01562dc6aa7+0x22cc8a)
503
    #27 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:322 (agatedb-6848b01562dc6aa7+0x2a0d6a)
504
    #28 std::panicking::try::do_call /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:381 (agatedb-6848b01562dc6aa7+0x203af9)
505
    #29 __rust_try test.cvxxcx5e-cgu.1:? (agatedb-6848b01562dc6aa7+0x20deab)
506
    #30 std::panicking::try /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:345 (agatedb-6848b01562dc6aa7+0x203357)
507
    #31 std::panic::catch_unwind /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396 (agatedb-6848b01562dc6aa7+0x2a324a)
508
    #32 std::thread::Builder::spawn_unchecked::{{closure}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473 (agatedb-6848b01562dc6aa7+0x22c9d3)
509
    #33 core::ops::function::FnOnce::call_once{{vtable-shim}} /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227 (agatedb-6848b01562dc6aa7+0x2a41eb)
510
    #34 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557b0b)
511
    #35 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1307 (agatedb-6848b01562dc6aa7+0x557c06)
512
    #36 std::sys::unix::thread::Thread::new::thread_start /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:71 (agatedb-6848b01562dc6aa7+0x519e03)
513

514
SUMMARY: ThreadSanitizer: data race /usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/alloc.rs:42 in std::sys::unix::alloc::<impl core::alloc::global::GlobalAlloc for std::alloc::System>::dealloc
515
==================

agatedb 中的 yatp 是否可以拿出来放到AgateOptions中

看起来，现在agatedb并没有类似rocksdb的Column family的东西，我一开始以为table是类似Column family，仔细看了一下代码，感觉table只是一个sst文件（没有compact）。

我的想法是，搞多个agatedb，每个相当于一个Column family

但是，现在每个agatedb都持有一份独立的yatp，开多个agatedb，就会有一堆线程。

我的想法是，把yatp放到参数中去，这样多个agatedb就可以共用一个yatp，不知道这个想法是否合理。

Feature Request: separate filter&index blocks and data blocks into two dedicated files, and SST supports range filter

When scan a range[start, end), LSM-tree based engine will create a iterator for each level and seek to the proper position for the corresponding SST file(whose range[smallest, biggest) cover [start, end) ). This will open the SST file and read some content of this SST even there is no key covered by range[start, end), for example the SST contains keys [a, e, f], and the scan range is [b, c).

This will open and read is not necessary if we can know this SST doesn't contain keys in the scan range. https://www.cs.cmu.edu/~pavlo/papers/mod601-zhangA-hm.pdf described a type named succinct range filter, which can achieve such goal. This can benefit even more when these SST files are stored in remote storage like cloud disk or distributed file system on the cloud.

Let's using the AWS EFS as an example, EFS has active and inactive tiers, it will move these inactive files to the inactive tier automatically to reduce the total storage cost. When the files in inactive tier are requested, EFS will move them to the active tier with some throughput fee. If we want to leverage such fundamental cloud storage, we need to consider how to make good use of its inactive tier to reduce the total storage cost, at the same time, we need to consider how to reduce the data transfer between its inactive tier and active tier.

If we separate the fitler&index block and the data blocks into 2 dedicated files, The index&filter files maybe accessed frequently, and they probably stay in the active tier of EFS, these long-time un-touched data files will stay in the inactive tier of EFS. This works well when the workload is pure point get, because bloom-filter will reduce these unnecessary touching of data files, but when the workload contains range queries, range filter make sense rather than bloom-filter.

Merge branch develop into master

Development Task

Merge develop into master part by part.

Development Progress

The pull requests merged at develop branch are as follows, I would try to apply them into master.

vlog
- #84 @zhangjinpeng1987 #143
- #88 @zhangjinpeng1987 #127
memtable
- #45 @GanZiheng #126
- #46 @GanZiheng #126
levels
- #47 @zhangjinpeng1987 #138
- #50 @zhangjinpeng1987 #148
- #52 @GanZiheng #129
- #54 @GanZiheng #129
- #75 @GanZiheng #129
- #73 @GanZiheng #129
- #77 @GanZiheng #129
- #78 @GanZiheng #129
- #101 @GanZiheng #129
txn
- #57 @GanZiheng #132
- #67 @GanZiheng
  - #136
  - #143
- #72 @GanZiheng #143
agate
- #58 @GanZiheng #127
- #62 @GanZiheng #127
- #60 @GanZiheng #123
- #87 @GanZiheng #170
- #91 @GanZiheng #170
bench
- #86 @GanZiheng #169
- #90 @GanZiheng #169
- #93 @GanZiheng #167
- #94 @GanZiheng #167
- #96 @GanZiheng #167
- #99 @GanZiheng #167
- #102 @GanZiheng #167

Developing roadmap

Stage 1: Finish basic implementation

Merge the develop branch to master, and reach the same test coverage ratio as bager
- #125
- introduce code-cov bot #131
Construct a comparable benchmark between RocksDB and agatedb
- port https://github.com/dgraph-io/badger-bench #169
- continuing benchmark #173

Stage 2: Integrate to TiKV

Stage 3: Compatibile to old TiKV

RocksDB SST format compatibility
Support upgrade from RocksDB directly(reading RocksDB manifest at initial)

Stage 4: Optimization

User timestamp support
Async IO support

Integrate with TiKV

Development Task

Integrate with TiKV (3-4 weeks)

Development Progress

TBD

Transaction 的 set(key,value) 不会增加版本号，当前进程读取是新数据，关闭进程后重新读取就会读取旧数据

读取都是用的

pub fn get(&self, key: impl Into<BytesMut>) -> Result<Value> {
    let key = key_with_ts(key.into(), std::u64::MAX);
    self.core.get(&key)
}

Proposal for new AgateIterator interface

The iterator trait can panic due to incorrect usage. I propose a safer design:

trait KeyValuePair {
    fn key(&self) -> &[u8];
    fn value(&self) -> Value;
}

trait Iterator {
    type KeyValuePair;

    fn next(&mut self) -> Result<KeyValuePair>;
    fn seek(&mut self) -> Result<KeyValuePair>;
    fn rewind(&mut self) -> Result<KeyValuePair>;
    fn current(&self) -> Option<KeyValuePair>;
}

KeyValuePair trait allows customization of key and value, so it's still lazily loaded. And key and value methods will not have to check for valid any more. Iterator doesn't have to store the error either.

This interface is for future refactor, doesn't have to be implemented in this PR.

Originally posted by @BusyJay in #30 (comment)

Stale README

Such as: https://github.com/tikv/agatedb/blob/49dc9b5d6038b5245042a10fa809d24243be7d2e/README.md?plain=1#L22C41-L22C48
the "develop" branch has not been updated in serval years.

Feature Request: cache system with precious memory usage limitation

Currently every sst is opened in mmap mode, we can't control the memory usage of agatedb with this mode, it will use as much as it can. This may cause OOM when agatedb embedded in other system (like in mongo's early days', its mmap engine caused many issues).

Implement a cache system that can control its total memory usage is very important for a storage engine, especially on the cloud environment where the resource is strictly limited. RocksDB's block-cache is good reference, it can limit the total memory usage and provide impressive access performance.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.