Segmentation fault when using `omnitrace` for generating instrumented binary about omnitrace HOT 8 CLOSED

This is a segfault in Dyninst (third-party library). I'll patch it in our submodule and also submit a patch upstream. From the looks of it, this is the offending line in dyninst:

            int          result = dwarf_cfi_addrframe(cfi_data[i], next_pc, &frame);

I suspect there is a problem with the pointer to the data in cfi_data[i]

from omnitrace.

I was not able to figure out what exactly causes the error from the error log.

The most relevant info is the backtrace to where the error occurred, e.g.:

Backtrace (demangled):
[PID=232867][TID=1][0/18] __sigaction +0x50
[PID=232867][TID=1][1/18] openbackend.constprop.0 +0x3f6
[PID=232867][TID=1][2/18] ebl_openbackend +0x35
[PID=232867][TID=1][3/18] __libdw_frame_at_address +0x28c
[PID=232867][TID=1][4/18] dwarf_cfi_addrframe +0x32
[PID=232867][TID=1][5/18] Dyninst::DwarfDyninst::DwarfFrameParser::getRegsForFunction(std::pair<unsigned long, unsigned long>, Dyninst::MachRegister, std::vector<Dyninst::VariableLocation, std::allocator<Dyninst::VariableLocation>>&, Dyninst::DwarfDyninst::FrameErrors_t&) +0x125
[PID=232867][TID=1][6/18] Dyninst::SymtabAPI::FunctionBase::expandLocation(Dyninst::VariableLocation const&, std::vector<Dyninst::VariableLocation, std::allocator<Dyninst::VariableLocation>>&) +0x115
[PID=232867][TID=1][7/18] Dyninst::SymtabAPI::FunctionBase::getFramePtr() +0x9e
[PID=232867][TID=1][8/18] Dyninst::SymtabAPI::DwarfWalker::getFrameBase() +0xba
[PID=232867][TID=1][9/18] Dyninst::SymtabAPI::DwarfWalker::parseSubprogram(Dyninst::SymtabAPI::DwarfWalker::inline_t) +0x38e
[PID=232867][TID=1][10/18] Dyninst::SymtabAPI::DwarfWalker::parse_int(Dwarf_Die, bool, bool) +0x98d
[PID=232867][TID=1][11/18] Dyninst::SymtabAPI::DwarfWalker::parse_int(Dwarf_Die, bool, bool) +0xa87
[PID=232867][TID=1][12/18] Dyninst::SymtabAPI::DwarfWalker::parse_int(Dwarf_Die, bool, bool) +0xa87
[PID=232867][TID=1][13/18] Dyninst::SymtabAPI::DwarfWalker::parseModule(Dwarf_Die, Dyninst::SymtabAPI::Module*&) +0x320
[PID=232867][TID=1][14/18] Dyninst::SymtabAPI::DwarfWalker::parse() [clone ._omp_fn.0] +0x14a
[PID=232867][TID=1][15/18] omp_fulfill_event +0x7be
[PID=232867][TID=1][16/18] pthread_condattr_setpshared +0x513
[PID=232867][TID=1][17/18] clone +0x44

In frame 5 (of 18) we see the call to Dyninst::DwarfDyninst::DwarfFrameParser::getRegsForFunction and then in frame 4, we see the call to dwarf_cfi_addrframe which is clearly called in the snippet above.

from omnitrace.

Can you turn up the verbosity to, say, level 4, (i.e. pass -v 4) and provide me with the last handful of log messages before the segfault. In the gist, it is currently this:

[omnitrace][exe] Finding instrumentation functions...

But there should be quite a bit more going on behind the scenes before the segfault.

from omnitrace.

@vincentadamthefirst I've been working on implementing some automated testing that uses thread and address sanitizers. I am seeing a lot of data races in Dyninst. You might want to try setting OMP_NUM_THREADS=1 in the environment and seeing if that fixes your issue.

from omnitrace.

@jrmadsen Thanks for looking into it.
I have executed omnitrace with -v 4 option and the (full) output can be seen in this gist: https://gist.github.com/vincentadamthefirst/ea39cc7af87dacaaef9a5b1c399db0dd

I also tried setting OMP_NUM_THREADS=1 which still resulted in a segmentation fault but got further in the instrumentation: https://gist.github.com/vincentadamthefirst/987baef411f78a7add2513073ea78a5b#file-omnitrace_error_verbose_omp_num_threads-txt-L336

from omnitrace.

I see from the backtrace you are picking up omnitrace's libdw but the containers libelf:

/home/vincent/omnitrace/install/lib/omnitrace/libdw-0.182.so
/.singularity.d/libs/libelf.so.1

You might want to try prefixing LD_LIBRARY_PATH with /home/vincent/omnitrace/install/lib/omnitrace
so that the omnitrace exe loads the libelf library installed by omnitrace in that folder. The dwarf parsing and elf parsing pretty much go hand-in-hand so there's a chance that the root cause is a weird ABI issue.

from omnitrace.

@vincentadamthefirst did you try the recommendation in the last comment?

from omnitrace.

@jrmadsen Sorry for answering so late, yes this did resolve my issues, thank you.

from omnitrace.