[BUG] Segmentation fault in Interruptible OpenMP about raft HOT 3 CLOSED

I ran the test in a loop (both, using --gtest_repeat=-1 and a bash loop), never triggered that on the host/conda. In your reproducer case, was getting it roughly in every 2 out of 3 tests.
The problem is more-or-less clear - the unspecified order of destruction across translation units and/or between thread-local and static variables. Perhaps, the inlined static/thread-local variables end up in different translation units depending on the library we link against?.. Or it could be some totally random unrelated thing.

from raft.

Thanks for the report! I could reproduce this as well with the same container and using libiomp5 (though I couldn't reproduce it with any of the openmp packages coming with conda). Also, with #1229 in place, the segfault does not happen.

For the record, here's the commands I used (within the raft root folder, using the same test from the raft's core test suite):

docker run -v `pwd`:/workspace/raft  --gpus all -it --rm nvcr.io/nvidia/pytorch:23.01-py3
...
apt update && apt install -y libgtest-dev libomp-dev gdb
cd /workspace/raft
nvcc -Icpp/include -Xcompiler=-fopenmp --std=c++17 cpp/test/core/interruptible.cu  -o test -lgtest -lgtest_main -lomp5
./test

Interestingly, I could only reproduce the error when running all the tests from cpp/test/core/interruptible.cu together, but couldn't when running any one of the tests individually using --gtest_filter. Also, the segmentation fault occurred at the very end, after gtest reported that all tests passed (this is expectable, because we assume the program crashes at the exit).

from raft.

FYI conda symlinks llvm openmp to libgomp: https://github.com/conda-forge/_openmp_mutex-feedstock/blob/main/recipe/build.sh#L12

They also build openmp with static libgcc and static libstdc++:
https://github.com/conda-forge/openmp-feedstock/blob/main/recipe/build-llvm-openmp.sh#L21

I understand the symlinking part but don't know if the static linking has anything to do with the segfault not showing up, curious if you know anything about it. Also did you run the reproducer multiple times? The error is a bit non deterministic and showed up ~1/3 times when I tested with libiomp5/libomp. So I'm wondering if the segfault not showing up with conda openmp was just a one off event.

from raft.