Error loading libmana.so about mana HOT 27 CLOSED

@marcpb94 , @JainTwinkle now has an internal version that may work on simple cases. But it probably won't work for all cases. At this time, she is passing a container with her solution to @karya0, who will then improve that to a robust solution. We truly are getting close now.

from mana.

Hi @marcpb94 (and @fyshhh),
The message means that it was looking for this file:
bin/../lib/dmtcp/libmana.so

However, LD_PRELOAD requires an absolute path (starting with '/'). For security reasons, it will refuse to load a relative path.

This appears to be a bug in how we configure MANA in CentOS. Could you do:

1. Please check if $MANA_ROOT/lib/dmtcp/libmana.so exists.
   1.a. If the file doesn't exist, please go back and do make -j mana at top level (one more time), and see if that fixes it.
   1.b. If the file does exist, then see step 2.
2. Could you report the steps that you used on CentOS that resulted in this problem that resulted in a relative path?
   2.a. If you want to dig more deeply on your own, you can look at $MANA_ROOT/src/dmtcp_launch.cpp to see how it sets LD_PRELOAD. You can add print statements or use GDB to see why it sets LD_PRELOAD badly. (And we'll also do that if we can reproduce what you saw.)

@marcpb94, and whatever happens, please let us know, so that we can update our documentation. Thanks.

@fyshhh,
Please continue to work with Marc to fix this configure bug. And based on what you discover, please update the documentation of CentOS. If you can't see the same bug on CentOS 7 on your side, please arrange for a screen-sharing session with @marcpb94.

from mana.

Hi,
Thanks for the quick response. I used the following commands:

git clone https://github.com/mpickpt/mana
cd mana
git submodule update --init --recursive
./configure-mana
make -j mana
bin/mana_coordinator
mpirun -np 4 bin/mana_launch mpi-proxy-split/test/mpi_hello_world.exe

Doing make -j mana a second time seemed to properly install libmana.so (and other missing libraries) in $MANA_ROOT/lib/dmtcp/, and the application seems to run without issues. I'll check that everything works tomorrow. Do I close this issue (since the specific error is solved), or should I keep it open for further problems that I encounter?

Thanks for the help!

from mana.

Hi,

Turns out that yesterday I tried to execute the "non-mana" compiled version of the examples (it surprisingly worked and performed checkpoints? Although they do not seem to recover successfully). When I try to execute the mana.exe versions of the examples, the applications get stuck and do not give any output whatsoever.

from mana.

@marcpb94 , if you're comfortable using email, you can reach me at gene with the d domain: ccs neu edu
I've now added a PR to fix a problem, so that you will no longer need to do make mana twice. Could you try using that PR? It might also fix a bug related to CentOS 7, but I didn't have a chance to test that.

@fyshhh , could you also try building MANA one more time? Two weeks ago, we changed to a new structure in the current default branch. I suspect that you'll need my latest PR to build MANA. Thanks for checking this.

from mana.

@marcpb94 and @fyshhh , I've now pushed in PR #135 . This is the PR that I created yesterday. Could each of you try MANA now, on the default branch, to see if it works correctly on CentOS 7 ? Whoever does it first, please report back the status here.

from mana.

Hi @gc00,

I just tried with the default branch and a single make -j mana is enough now. I still get stuck executing applications unfortunately. Would be interesting to see if @fyshhh can replicate this issue. I will continue looking into this on monday, thank you both!

from mana.

Thanks @marcpb94 .

@fyshhh , please see if you can reproduce the bug in CentOS 7/MANA. If you cannot reproduce the bug, then please arrange with @marcpb94 for a screen-sharing session. It's probably a 9 hour difference in time zones (Western Europe and California).

from mana.

@gc00 and @marcpb94 - interesting, I'm seeing this bug on CentOS 7 as well. It seems to be a segmentation fault that occurs when initialising the lower half. I'll see if I can roll out a fix by this weekend.

from mana.

@JainTwinkle , This seems to be the same bug that you were looking at today.

@fyshhh , Can you go through the git history to see which commit caused this to break? I wonder if it's related to us now using the feature/dmtcp-master branch. Thanks for looking into this.

from mana.

@fyshhh, And if you find that the bug appeared when we switched to feature/dmtcp-master, then please go through those commits, and try cherry-picking my latest PR #135 at each step. Probably some of those commits are fixed by PR #135.

from mana.

@gc00 It may be the same bug. But, we need the backtrace for the confirmation.

@fyshhh and @marcpb94, could either of you check if it's reaching the segfault handler and getting stuck in an infinite loop?
The top command should also show you 100% CPU usage if the program is in an infinite loop.

@gc00, maybe we should print it on the terminal whenever the segfault handler is called. Currently, I see no message stating that a segfault has happened, and I am supposed to attach gdb.

from mana.

@JainTwinkle yes, it was reaching the segfault handler. This is the backtrace when run on the HEAD commit of feature/dmtcp-master:

#0  segvfault_handler (signum=11, siginfo=0x7fff0c629130, context=0x7fff0c629000) at mpi_plugin.cpp:106
#1  <signal handler called>
#2  0x000000000e20ff3d in getenv ()
#3  0x000000000e26c761 in _dl_non_dynamic_init ()
#4  0x000000000e26d0f4 in __libc_init_first ()
#5  0x000000000e2092ce in generic_start_main ()
#6  0x000000000e2095d1 in __libc_start_main ()
#7  0x00007f0ebd82d016 in initializeLowerHalf () at split_process.cpp:469
#8  0x00007f0ebd82af7e in splitProcess () at split_process.cpp:103
#9  0x00007f0ebd812df9 in mpi_plugin_event_hook (event=DMTCP_EVENT_INIT, data=0x0) at mpi_plugin.cpp:237
#10 0x00007f0ebcb78c59 in dmtcp::PluginManager::eventHook (event=DMTCP_EVENT_INIT, data=0x0) at pluginmanager.cpp:98
#11 0x00007f0ebcb6cec0 in dmtcp_initialize_entry_point () at dmtcpworker.cpp:292
#12 0x00007f0ebdacd9c3 in call_init (env=0x7fff0c62b128, argv=0x7fff0c62b118, argc=1, l=<optimized out>)
    at dl-init.c:82
#13 _dl_init (main_map=0x7f0ebdce1150, argc=1, argv=0x7fff0c62b118, env=0x7fff0c62b128) at dl-init.c:131
#14 0x00007f0ebdabf17a in _dl_start_user ()
#15 0x0000000000000001 in ?? ()
#16 0x00007fff0c62ba9e in ?? ()
#17 0x0000000000000000 in ?? ()

from mana.

@gc00 @fyshhh, I'm seeing the same backtrace.

from mana.

@gc00 @JainTwinkle the last working commit is [6bf4b7a89e69aeecfce3e6e0c575752d513e7168], and it appears the big group of commits in 6bf4b7a..73d877e is what's causing the bug. I can't go through each of them individually - MANA will fail to compile at some commits (probably because they represent one big change). Anyway, it does occur when we switched to using the dmtcp submodule.

from mana.

@fyshhh Thanks! I'm also looking into it.

For me, if I run mana_launch (actual command) under gdb, mpi_hello_world works fine (at least for a single rank). So, I'm guessing that it can be some subtle memory corruption.

from mana.

Backtrace with more info:

#0  segvfault_handler (signum=11, siginfo=0x7ffe9159efb0, context=0x7ffe9159ee80) at mpi_plugin.cpp:112
#1  <signal handler called>
#2  getenv (name=0xe385f55 "_BIND_NOW") at getenv.c:88
#3  0x000000000e256db1 in _dl_non_dynamic_init () at dl-support.c:285
#4  0x000000000e257744 in __libc_init_first (argc=1, argv=0x7ffe915a0fa8, envp=0x7ffe915a0fb8) at ../csu/init-first.c:82
#5  0x000000000e1f429e in generic_start_main (main=main@entry=0xe00849d <main>, argc=argc@entry=1, argv=argv@entry=0x7ffe915a0fa8, 
    init=init@entry=0xe1f4bc0 <__libc_csu_init>, fini=fini@entry=0xe1f4c50 <__libc_csu_fini>, rtld_fini=0x0, 
    stack_end=stack_end@entry=0x7ffe915a0f98) at ../csu/libc-start.c:205
#6  0x000000000e1f45a1 in __libc_start_main (main=0xe00849d <main>, argc=1, argv=0x7ffe915a0fa8, init=0xe1f4bc0 <__libc_csu_init>, 
    fini=0xe1f4c50 <__libc_csu_fini>, rtld_fini=<optimized out>, stack_end=0x7ffe915a0f98) at ../sysdeps/x86/libc-start.c:38
#7  0x00007f6b90c1773f in initializeLowerHalf () at split_process.cpp:465
#8  0x00007f6b90c15596 in splitProcess () at split_process.cpp:104
#9  0x00007f6b90bfd1cd in mpi_plugin_event_hook (event=DMTCP_EVENT_INIT, data=0x0) at mpi_plugin.cpp:243
#10 0x00007f6b908f9eaf in dmtcp::PluginManager::eventHook (event=DMTCP_EVENT_INIT, data=0x0) at pluginmanager.cpp:98
#11 0x00007f6b908edf3c in dmtcp_initialize_entry_point () at dmtcpworker.cpp:292
#12 0x00007f6b90a999c3 in call_init (env=0x7ffe915a0fb8, argv=0x7ffe915a0fa8, argc=1, l=<optimized out>) at dl-init.c:82
#13 _dl_init (main_map=0x7f6b90cad150, argc=1, argv=0x7ffe915a0fa8, env=0x7ffe915a0fb8) at dl-init.c:131
#14 0x00007f6b90a8b17a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2

from mana.

@gc00 and @fyshhh, I figured the issue.

Short story: Initially, one environment variable's location is in the heap, and later, the heap gets shrunk/unmapped. So, the env variable's address becomes invalid. On my machine, it's the PATH env variable.

Long story:

*ep = 0x1b9d010 "PATH=/home/twinkle/mana-feature/bin:/opt/rh/devtoolset-9/root/usr/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr"...

PATH's address is in the heap:

 0x1b9d000          0x1bbe000    0x21000        0x0 [heap]

Later, the heap gets unmapped and the env pointer still points to the heap. So, the program segfaults.

The following is the relevant code from glibc: (see my comment @ L273 and L277)

272       _dl_verbose = *(getenv ("LD_WARN") ?: "") == '\0' ? 0 : 1;
273        <-------- Till this point, PATH's location is in the heap (mapped and valid)
274       /* Set up the data structures for the system-supplied DSO early,
275          so they can influence _dl_init_paths.  */
276       setup_vdso (NULL, NULL); 
277       <--------- At this point, the heap is gone, but PATH still points to heap! (unmapped and invalid)
278       /* Initialize the data structures for the search paths for shared
279          objects.  */
280       _dl_init_paths (getenv ("LD_LIBRARY_PATH"));

I still need to figure out why the heap gets de-allocated and how we can make sure that the PATH stays valid. I can think of a few solutions but I'd like to gather more information at this point.

from mana.

@gc00 @fyshhh,

I looked further into this and also compared GDB (successful) vs. non-GDB (failed) runs. It's the setup_vdso function in the glibc that clears out space for the arena and unmaps the heap in the failed case. I have updated my previous comment to show the correct line number in the libc's code.

Following is the backtrace to show when the heap set up by MANA gets unmapped:

#0  munmap () at ../sysdeps/unix/syscall-template.S:81
#1  0x000000000e2963f6 in __wrap___munmap (addr=0x604000, len=60801024) at munmap.c:107
#2  0x000000000e2171db in new_heap (size=135168, size@entry=3401, top_pad=<optimized out>) at arena.c:548
#3  0x000000000e217731 in _int_new_arena (size=1161) at arena.c:736
#4  arena_get2 (size=size@entry=1161, avoid_arena=avoid_arena@entry=0x0, a_tsd=<optimized out>) at arena.c:971
#5  0x000000000e21de42 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:3211
#6  0x000000000e283750 in _dl_new_object (realname=realname@entry=0xe396e04 "", libname=libname@entry=0xe396e04 "", type=type@entry=1, loader=loader@entry=0x0, 
    mode=mode@entry=0, nsid=nsid@entry=0) at dl-object.c:76
#7  0x000000000e256950 in setup_vdso (main_map=0x0, first_preload=0x0) at setup-vdso.h:32
#8  _dl_non_dynamic_init () at dl-support.c:276
#9  0x000000000e257744 in __libc_init_first (argc=1, argv=0x7ffd8cddd4f8, envp=0x7ffd8cddd508) at ../csu/init-first.c:82
#10 0x000000000e1f429e in generic_start_main (main=main@entry=0xe00849d <main>, argc=argc@entry=1, argv=argv@entry=0x7ffd8cddd4f8, 
    init=init@entry=0xe1f4bc0 <__libc_csu_init>, fini=fini@entry=0xe1f4c50 <__libc_csu_fini>, rtld_fini=0x0, stack_end=stack_end@entry=0x7ffd8cddd4e8)
    at ../csu/libc-start.c:205
#11 0x000000000e1f45a1 in __libc_start_main (main=0xe00849d <main>, argc=1, argv=0x7ffd8cddd4f8, init=0xe1f4bc0 <__libc_csu_init>, fini=0xe1f4c50 <__libc_csu_fini>, 
    rtld_fini=<optimized out>, stack_end=0x7ffd8cddd4e8) at ../sysdeps/x86/libc-start.c:38
#12 0x00007f78acf5d74e in initializeLowerHalf () at split_process.cpp:466

Non-GDB: The procmaps below are self-explanatory:

BEFORE setup_vdso:
          Start Addr           End Addr       Size     Offset objfile
            0x404000           0x405000     0x1000     0x3000 /home/twinkle/mana-new/mpi-proxy-split/test/mpi_hello_world.mana.exe
            0xc9e000           0xcbf000    0x21000        0x0 [heap]  <-- PATH's location
           0xe000000          0xe001000     0x1000        0x0 /home/twinkle/mana-feature/bin/lh_proxy

DURING setup_vdso:
          Start Addr           End Addr       Size     Offset objfile
            0x404000           0x405000     0x1000     0x3000 /home/twinkle/mana-new/mpi-proxy-split/test/mpi_hello_world.mana.exe
            0x604000          0x8604000  0x8000000        0x0 [heap]
           0xe000000          0xe001000     0x1000        0x0 /home/twinkle/mana-feature/bin/lh_proxy

GDB (running mana_launch/dmtcp_launch under gdb directly):

BEFORE setup_vdso:
          Start Addr           End Addr       Size     Offset objfile
            0x404000           0x405000     0x1000     0x3000 /home/twinkle/mana-feature/mpi-proxy-split/test/mpi_hello_world.mana.exe
            0x405000           0x426000    0x21000        0x0 [heap]  <== PATH's location

DURING setup_vdso:
          Start Addr           End Addr       Size     Offset objfile
            0x404000           0x405000     0x1000     0x3000 /home/twinkle/mana-feature/mpi-proxy-split/test/mpi_hello_world.mana.exe
            0x405000           0x426000    0x21000        0x0 [heap]  <== PATH's location
            0x604000          0x8604000  0x8000000        0x0 

One thing to note is that in the failed case, the heap's location (where PATH env var exists) is not right after the data section of the executable, and it changes in each run. However, in GDB, it is right after the data section, and it doesn't overlap with the arena. So. the heap and PATH's location stay valid after the setup_vdso function.

I am not sure but it seems that we are messing up the sbrk for the upper half. It is supposed to be right after the data segment (as we can see in the GDB's case).

from mana.

@marcpb94 ,
@JainTwinkle and I talked today. She is guessing that she will have a fix in place in about 2 or 3 days. (See the diagnosis in her earlier e-mail.) The issue arose when MANA switched to a new architecture about 3 weeks ago.

The new architecture was required so that MANA would inherit the latest version of DMTCP. (DMTCP is the underlying platform for MANA.) Prior to that, the student who developed the original MANA in his thesis had frozen a version of DMTCP, in order to have a stable platform for his thesis. We had to significantly change the MANA architecture to make DMTCP a git submodule in the MANA github repo.

We had tested the new architecture on the Cori supercomputer, but apparently not on CentOS. Sorry for the rough ride.

from mana.

No worries @gc00, I'm glad it is getting attention and fixed soon! I'm currently working with the older version (with the frozen version of DMTCP), and we wanted to move to the new version to see if some of the issues we experienced are fixed (I saw some promising bug fixes from one or two months ago).

from mana.

Hi, any updates on the fix?

from mana.

@gc00 @marcpb94
I found the root cause of the problem. It's the findLHMemRange in split_process.cpp that gives the wrong range to lower-half's mmap at launch and this range (lh_memRange = {start = 0x600000, end = 0x80600000}) interferes with upper-half's mapped regions. If I manually fix this range (that does not overlap with upper-half regions), the program executes fine. I'm currently working on fixing the findLHMemRange code. I'm confident that fixing this function will resolve all subsequent memory corruption issues.

from mana.

Hi @gc00 @karya0

I am able to launch, checkpoint, and restart single rank mpi_hello_world on CentOS by fixing findLHMemRange on top of my other PR for copying a variable number of lower-half regions. However, for multiple ranks, all the restarted processes wait forever for all the other ranks to finish the restore task. Have you seen this issue before? It would be helpful if we could have a debugging session.

Backtrace:

(gdb) thread apply all bt

Thread 2 (Thread 0x7ff4dfff7700 (LWP 30647)):
#0  0x00007ff4e3172ddd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ff4e361bde2 in dmtcp::ConnectionList::sendReceiveMissingFds (this=0x7ff4e36d3608) at ipc/connectionlist.cpp:596
#2  0x00007ff4e361b2c9 in dmtcp::ConnectionList::refill (this=0x7ff4e36d3608, isRestart=true) at ipc/connectionlist.cpp:467
#3  0x00007ff4e3661186 in dmtcp::SocketConnList::refill (this=0x7ff4e36d3608, isRestart=true) at ipc/socket/socketconnlist.cpp:234
#4  0x00007ff4e36618f0 in dmtcp::SocketConnList::restartRefill () at ipc/socket/socketconnlist.h:47
#5  0x00007ff4e3660ac4 in dmtcp_SocketConnList_EventHook (event=DMTCP_EVENT_RESTART, data=0x0) at ipc/socket/socketconnlist.cpp:78
#6  0x00007ff4e350e14c in dmtcp::PluginManager::eventHook (event=DMTCP_EVENT_RESTART, data=0x0) at pluginmanager.cpp:153
#7  0x00007ff4e350332f in dmtcp::DmtcpWorker::postRestart (ckptReadTime=0) at dmtcpworker.cpp:517
#8  0x00007ff4e3518737 in dmtcp::ThreadList::waitForAllRestored (thread=0x7ff4e345c008) at threadlist.cpp:663
#9  0x00007ff4e35171b1 in checkpointhread (dummy=0x0) at threadlist.cpp:394
#10 0x00007ff4e3519f25 in thread_start (arg=0x7ff4e345c008) at threadwrappers.cpp:108
#11 0x00007ff4e2446ea5 in start_thread (arg=0x7ff4dfff7700) at pthread_create.c:307
#12 0x00007ff4e317db0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7ff4e36d8780 (LWP 30599)):
#0  0x00007ff4e244cb3b in futex_abstimed_wait (cancel=true, private=<optimized out>, abstime=0x0, expected=0, futex=0x7ff4e35940a0 <semWaitForCkptThreadSignal>) at ../nptl/sysdeps/unix/sysv/linux/sem_waitcommon.c:43
#1  do_futex_wait (sem=sem@entry=0x7ff4e35940a0 <semWaitForCkptThreadSignal>, abstime=0x0) at ../nptl/sysdeps/unix/sysv/linux/sem_waitcommon.c:223
#2  0x00007ff4e244cbcf in __new_sem_wait_slow (sem=0x7ff4e35940a0 <semWaitForCkptThreadSignal>, abstime=0x0) at ../nptl/sysdeps/unix/sysv/linux/sem_waitcommon.c:292
#3  0x00007ff4e244cc6b in __new_sem_wait (sem=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/sem_wait.c:28
#4  0x00007ff4e35187b8 in dmtcp::ThreadList::waitForAllRestored (thread=0x7ff4e345d008) at threadlist.cpp:684
#5  0x00007ff4e35185a7 in stopthisthread (signum=12) at threadlist.cpp:630
#6  <signal handler called>
#7  0x00007ff4e31449fd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#8  0x00007ff4e3144894 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:137
#9  0x0000000000401367 in main (argc=1, argv=0x7ffc749e8a48) at mpi_hello_world.c:49

from mana.

Hi @marcpb94

Could you please try PR #160 on top of MANA's main branch and let us know if MANA works on CentOS for you?
Thanks!

from mana.

Hi @JainTwinkle

It seems to be working now! I tried with mpi_hello_world with 1 rank and a heat distribution application (that we usually use for testing) with 4 ranks.

However, there seems to be an issue we already noticed in the old version of MANA, and we were hoping it was fixed in the new version. It seems to happen with our heat distribution application, I attach the source code so that you are able to reproduce the problem. heatdis.zip

The issue is that when checkpointing with relatively short intervals(and letting it execute for a few minutes), the execution eventually encounters a deadlock in the pre-checkpoint phase. Specifically, at least one of the MPI processes gets stuck in the drainSendRecv() function in the DMTCP_EVENT_PRECHECKPOINT case from mpi_plugin_event_hook().

Should we continue in this thread? Or should I make another issue?

Thanks,

Marc

from mana.

Hi @marcpb94

I've created a new Github issue (#161) to discuss the deadlock issue. Let's discuss it there!

from mana.