Comments (7)
PlasmaPower
commented on May 17, 2024
The simplest solution would be to add a Drop impl to each of these types that calls try_pop in a loop, but atomic operations aren't strictly necessary, as we're guaranteed at the time drop is called that only one thread is accessing the structure.
I'm running into a related problem - a Into<Vec<T>> or more likely Into<Iterator<T>> implementation is missing. The implementation should be similar.
from crossbeam.
jeehoonkang
commented on May 17, 2024
For ms queue & segmented queue, even if every element is popped, I think the sentinel node is still not deallocated. Note that the first sentinel node is created at, e.g., https://github.com/aturon/crossbeam/blob/master/src/sync/ms_queue.rs#L65, and later the just-popped element serves as the new sentinel. More concretely, I think the first sentinel node is leaked in the following code.
extern crate crossbeam;
fn main() {
let queue = crossbeam::sync::MsQueue<i32>::new();
}
The Treiber stack does not suffer from this problem, because null serves as the sentinel in the stack.
from crossbeam.
Vtec234
commented on May 17, 2024
The above is correct. The sentinel is leaked, as confirmed by valgrind for MsQueue:
==14055== 24 bytes in 1 blocks are definitely lost in loss record 1 of 7
==14055== at 0x4C28EBF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==14055== by 0x14C66C: alloc::heap::allocate::h8f177ce216eebe59 (heap.rs:59)
==14055== by 0x14C5E0: alloc::heap::exchange_malloc::h3e6e0f3cc2ffa7ba (heap.rs:138)
==14055== by 0x15CA6F: new<crossbeam::sync::ms_queue::Node<i64>> (boxed.rs:235)
==14055== by 0x15CA6F: _$LT$crossbeam..mem..epoch..Owned$LT$T$GT$$GT$::new::h7e0b820346f52ee7 (mod.rs:155)
==14055== by 0x160A90: _$LT$crossbeam..sync..ms_queue..MsQueue$LT$T$GT$$GT$::new::hed250e14ac3da10e (ms_queue.rs:65)
==14055== by 0x164FF6: crossbeam::sync::ms_queue::test::leak::h0b93c2b69d77a47c (ms_queue.rs:321)
==14055== by 0x18E15E: {{closure}} (lib.rs:1360)
==14055== by 0x18E15E: _$LT$F$u20$as$u20$test..FnBox$LT$T$GT$$GT$::call_box::hcd22cc8d960226e1 (lib.rs:140)
==14055== by 0x1DB00A: __rust_maybe_catch_panic (lib.rs:98)
==14055== by 0x18260A: try<(),std::panic::AssertUnwindSafe<closure>> (panicking.rs:436)
==14055== by 0x18260A: catch_unwind<std::panic::AssertUnwindSafe<closure>,()> (panic.rs:361)
==14055== by 0x18260A: {{closure}} (lib.rs:1305)
==14055== by 0x18260A: call_once<(),closure> (panic.rs:296)
==14055== by 0x18260A: std::panicking::try::do_call::h515bb5f8d6547fa7 (panicking.rs:460)
==14055== by 0x1DB00A: __rust_maybe_catch_panic (lib.rs:98)
==14055== by 0x1891A6: try<(),std::panic::AssertUnwindSafe<closure>> (panicking.rs:436)
==14055== by 0x1891A6: catch_unwind<std::panic::AssertUnwindSafe<closure>,()> (panic.rs:361)
==14055== by 0x1891A6: {{closure}}<closure,()> (mod.rs:357)
==14055== by 0x1891A6: _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h52f58ba131328272 (boxed.rs:605)
==14055== by 0x1D2F14: call_once<(),()> (boxed.rs:615)
==14055== by 0x1D2F14: start_thread (thread.rs:21)
==14055== by 0x1D2F14: std::sys::imp::thread::Thread::new::thread_start::h76badbf9b0ecaf58 (thread.rs:84)
and SegQueue:
==13047== 280 bytes in 1 blocks are definitely lost in loss record 6 of 7
==13047== at 0x4C28EBF: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==13047== by 0x14CDBC: alloc::heap::allocate::h8f177ce216eebe59 (heap.rs:59)
==13047== by 0x14CD30: alloc::heap::exchange_malloc::h3e6e0f3cc2ffa7ba (heap.rs:138)
==13047== by 0x15D70F: new<crossbeam::sync::seg_queue::Segment<u32>> (boxed.rs:235)
==13047== by 0x15D70F: _$LT$crossbeam..mem..epoch..Owned$LT$T$GT$$GT$::new::h9528934703fbb5fa (mod.rs:155)
==13047== by 0x169E43: _$LT$crossbeam..sync..seg_queue..SegQueue$LT$T$GT$$GT$::new::h5cf7a5734d4619f3 (seg_queue.rs:61)
==13047== by 0x16B470: crossbeam::sync::seg_queue::test::leak::h9cbcb80048d7c26f (seg_queue.rs:136)
==13047== by 0x18F3FE: {{closure}} (lib.rs:1360)
==13047== by 0x18F3FE: _$LT$F$u20$as$u20$test..FnBox$LT$T$GT$$GT$::call_box::hcd22cc8d960226e1 (lib.rs:140)
==13047== by 0x1DC2AA: __rust_maybe_catch_panic (lib.rs:98)
==13047== by 0x1838AA: try<(),std::panic::AssertUnwindSafe<closure>> (panicking.rs:436)
==13047== by 0x1838AA: catch_unwind<std::panic::AssertUnwindSafe<closure>,()> (panic.rs:361)
==13047== by 0x1838AA: {{closure}} (lib.rs:1305)
==13047== by 0x1838AA: call_once<(),closure> (panic.rs:296)
==13047== by 0x1838AA: std::panicking::try::do_call::h515bb5f8d6547fa7 (panicking.rs:460)
==13047== by 0x1DC2AA: __rust_maybe_catch_panic (lib.rs:98)
==13047== by 0x18A446: try<(),std::panic::AssertUnwindSafe<closure>> (panicking.rs:436)
==13047== by 0x18A446: catch_unwind<std::panic::AssertUnwindSafe<closure>,()> (panic.rs:361)
==13047== by 0x18A446: {{closure}}<closure,()> (mod.rs:357)
==13047== by 0x18A446: _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h52f58ba131328272 (boxed.rs:605)
==13047== by 0x1D41B4: call_once<(),()> (boxed.rs:615)
==13047== by 0x1D41B4: start_thread (thread.rs:21)
==13047== by 0x1D41B4: std::sys::imp::thread::Thread::new::thread_start::h76badbf9b0ecaf58 (thread.rs:84)
It should be relatively simple to free them in a Drop implementation, it just requires an additional explicit memory free for the sentinel after freeing the valid nodes.
from crossbeam.
fulmicoton
commented on May 17, 2024
It hit us in tantivy. We use MsQueue as a very small pool of searcher. Each searcher retains a bunch of Mmap object. Not dropping these object prevented munmap from being ever called.
Our client-side fix is here
quickwit-oss/tantivy@15fc049
and is similar to what is suggested by @apasel422 .
from crossbeam.
commented on May 17, 2024
@fulmicoton Ouch, sorry for the bug!
Unfortunately, that fix does not completely solve the problem. One allocated sentinel node still gets leaked. After popping all nodes, the last remaining step is something like:
drop(Vec::from_raw_parts(self.head.load(Relaxed, &guard).unwrap().as_raw(), 0, 1);Someone interested in submitting a PR?
from crossbeam.
fulmicoton
commented on May 17, 2024
@stjepang no problem, it was not too difficult to locate the issue !
from crossbeam.
fulmicoton
commented on May 17, 2024
@stjepang awesome !
from crossbeam.
Related Issues (20)
- Can we avoid to call `try_advance` in some senarios? [crossbeam-epoch] HOT 2
- Epoch memory reclamation algorithm HOT 2
- consider adding `is_disconnected` method to Receiver type HOT 1
- Scaling receiver counter up/down based on demand
- `select!` macro and auto-complete HOT 1
- [crossbeam-epoch] false sharing issue for "epoch: AtomicEpoch" field in "Local" struct
- crossbeam-skiplist bug HOT 3
- epoch `fetch_update` not working as expected HOT 2
- Crossbeam channel slower than Go channels with input data delay HOT 5
- Questions about "crossbeam_epoch" HOT 10
- crossbeam channel is very slow if restricted to a single core HOT 4
- Use cfg(target_has_atomic) instead of hard coding which targets support atomics HOT 7
- Donating Funds to Crossbeam HOT 1
- performance issues with memory ordering
- Panicked during `epoch::pin()` HOT 9
- Implement fmt::Display for CachePadded where T: Display HOT 1
- Thread-safe, shared-state concurrency with `WaitGroup` HOT 2
- crossbeam-utils fails to compile on esp32* targets starting Rust 1.74 HOT 3
- provide a scoped thread start taht returns result HOT 1
- crossbeam::channel::Receiver::try_recv can block forever if sending thread is blocked HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from crossbeam.