Comments (7)
dotnwat
commented on July 17, 2024
Getting a stack trace is proving difficult. Catching the assertion with a busy loop so we can attach GBD appears to cause the problem to not occur. On the other hand, using the execinfo backtrace routine to grab the backtrace explicitly on the condition reports that the backtrace size is zero. So far no luck getting a backtrace.
from legion.
dotnwat
commented on July 17, 2024
Ahh, we lucked out. Here are the backtraces for all the threads. There are two processes. Thread 7 in the first proc is the one spinning right before it fails the assertion.
(lldb) p owner
(unsigned int) $3 = 1
I'm not sure what the value of gasnet_mynode() is.
first proc
* thread #1: tid = 0x62867, 0x00007fff8f4a848a libsystem_kernel.dylib`__semwait_signal + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
* frame #0: 0x00007fff8f4a848a libsystem_kernel.dylib`__semwait_signal + 10
frame #1: 0x00007fff9288df5d libsystem_c.dylib`nanosleep + 199
frame #2: 0x00007fff9288ddbd libsystem_c.dylib`sleep + 42
frame #3: 0x000000010bfc87e4 tester_io`Realm::RuntimeImpl::run(this=0x00007ffb984052a0, task_id=0, style=ONE_TASK_ONLY, args=0x0000000000000000, arglen=0, background=false) + 3940 at runtime_impl.cc:1103
frame #4: 0x000000010bfc7870 tester_io`Realm::Runtime::run(this=0x00007fff53da7438, task_id=0, style=ONE_TASK_ONLY, args=0x0000000000000000, arglen=0, background=false) + 80 at runtime_impl.cc:138
frame #5: 0x000000010c574805 tester_io`LegionRuntime::HighLevel::Runtime::start(argc=1, argv=0x00007fff53da7d38, background=false) + 7797 at runtime.cc:15685
frame #6: 0x000000010c01764a tester_io`LegionRuntime::HighLevel::HighLevelRuntime::start(argc=3, argv=0x00007fff53da7d28, background=false) + 42 at legion.cc:3671
frame #7: 0x000000010be7a390 tester_io`main(argc=3, argv=0x00007fff53da7d28) + 352 at tester_io.cc:244
frame #8: 0x00007fff93c255c9 libdyld.dylib`start + 1
frame #9: 0x00007fff93c255c9 libdyld.dylib`start + 1
thread #2: tid = 0x62873, 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #0: 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff90409e0c libsystem_pthread.dylib`_pthread_cond_wait + 693
frame #2: 0x000000010beba683 tester_io`IncomingMessageManager::get_messages(this=0x00007ffb986009e0, sender=0x000000010d2a4ea4, wait=true) + 131 at activemsg.cc:686
frame #3: 0x000000010beba801 tester_io`Realm::HandlerThread::thread_main(this=0x00007ffb98600ab0) + 49 at activemsg.cc:725
frame #4: 0x000000010bf66fbc tester_io`Realm::PreemptableThread::thread_entry(data=0x00007ffb98600ab0) + 156 at proc_impl.cc:1039
frame #5: 0x00007fff90409268 libsystem_pthread.dylib`_pthread_body + 131
frame #6: 0x00007fff904091e5 libsystem_pthread.dylib`_pthread_start + 176
frame #7: 0x00007fff9040741d libsystem_pthread.dylib`thread_start + 13
thread #3: tid = 0x62875, 0x00007fff8f4a83fa libsystem_kernel.dylib`__select + 10
frame #0: 0x00007fff8f4a83fa libsystem_kernel.dylib`__select + 10
frame #1: 0x000000010c781331 tester_io`inputWaiting(unsigned int) + 161
frame #2: 0x000000010c77e31e tester_io`AMUDP_SPMDHandleControlTraffic + 78
frame #3: 0x000000010c778b0d tester_io`AM_Poll + 77
frame #4: 0x000000010c70a392 tester_io`gasnetc_AMPoll + 50
frame #5: 0x000000010bebb941 tester_io`do_some_polling() [inlined] gasneti_AMPoll + 5 at gasnet_help.h:597
frame #6: 0x000000010bebb93c tester_io`do_some_polling() [inlined] gasnet_AMPoll at gasnet_help.h:712
frame #7: 0x000000010bebb93c tester_io`do_some_polling() + 28 at activemsg.cc:2084
frame #8: 0x000000010bebbd56 tester_io`gasnet_poll_thread_loop(data=0x0000000000000000) + 38 at activemsg.cc:2100
frame #9: 0x00007fff90409268 libsystem_pthread.dylib`_pthread_body + 131
frame #10: 0x00007fff904091e5 libsystem_pthread.dylib`_pthread_start + 176
frame #11: 0x00007fff9040741d libsystem_pthread.dylib`thread_start + 13
thread #4: tid = 0x62879, 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10 [0/254]
frame #0: 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff90409e0c libsystem_pthread.dylib`_pthread_cond_wait + 693
frame #2: 0x000000010bea4ace tester_io`LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(this=0x00007ffb98404c70, dma_thread=0x00007ffb98404d10, wait_on_empty=true) + 1902 at channel.h:1081
frame #3: 0x000000010be95709 tester_io`LegionRuntime::LowLevel::DMAThread::dma_thread_loop(this=0x00007ffb98404d10) + 1129 at channel.cc:1448
frame #4: 0x000000010bef043d tester_io`LegionRuntime::LowLevel::DMAThread::start(arg=0x00007ffb98404d10) + 29 at channel.h:977
frame #5: 0x00007fff90409268 libsystem_pthread.dylib`_pthread_body + 131
frame #6: 0x00007fff904091e5 libsystem_pthread.dylib`_pthread_start + 176
frame #7: 0x00007fff9040741d libsystem_pthread.dylib`thread_start + 13
thread #5: tid = 0x6287b, 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #0: 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff90409e0c libsystem_pthread.dylib`_pthread_cond_wait + 693
frame #2: 0x000000010bea4ace tester_io`LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(this=0x00007ffb98404c70, dma_thread=0x00007ffb98405b20, wait_on_empty=true) + 1902 at channel.h:1081
frame #3: 0x000000010be95709 tester_io`LegionRuntime::LowLevel::DMAThread::dma_thread_loop(this=0x00007ffb98405b20) + 1129 at channel.cc:1448
frame #4: 0x000000010bef043d tester_io`LegionRuntime::LowLevel::DMAThread::start(arg=0x00007ffb98405b20) + 29 at channel.h:977
frame #5: 0x00007fff90409268 libsystem_pthread.dylib`_pthread_body + 131
frame #6: 0x00007fff904091e5 libsystem_pthread.dylib`_pthread_start + 176
frame #7: 0x00007fff9040741d libsystem_pthread.dylib`thread_start + 13
thread #6: tid = 0x6287c, 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #0: 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff90409e0c libsystem_pthread.dylib`_pthread_cond_wait + 693
frame #2: 0x000000010bea4ace tester_io`LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(this=0x00007ffb98404c70, dma_thread=0x00007ffb98405f30, wait_on_empty=true) + 1902 at channel.h:1081
frame #3: 0x000000010be95709 tester_io`LegionRuntime::LowLevel::DMAThread::dma_thread_loop(this=0x00007ffb98405f30) + 1129 at channel.cc:1448
frame #4: 0x000000010bef043d tester_io`LegionRuntime::LowLevel::DMAThread::start(arg=0x00007ffb98405f30) + 29 at channel.h:977
frame #5: 0x00007fff90409268 libsystem_pthread.dylib`_pthread_body + 131
frame #6: 0x00007fff904091e5 libsystem_pthread.dylib`_pthread_start + 176
frame #7: 0x00007fff9040741d libsystem_pthread.dylib`thread_start + 13
thread #7: tid = 0x6287e, 0x000000010bf3d3bf tester_io`Realm::GenEventImpl::check_for_catchup(this=0x00007ffb9a091c50, implied_trigger_gen=209875311) + 607 at event_impl.cc:517
frame #0: 0x000000010bf3d3bf tester_io`Realm::GenEventImpl::check_for_catchup(this=0x00007ffb9a091c50, implied_trigger_gen=209875311) + 607 at event_impl.cc:517
frame #1: 0x000000010bfc9f5a tester_io`Realm::RuntimeImpl::get_genevent_impl(this=0x00007ffb984052a0, e=(id = 545260288, gen = 209875312)) + 330 at runtime_impl.cc:1275
frame #2: 0x000000010bfc9da4 tester_io`Realm::RuntimeImpl::get_event_impl(this=0x00007ffb984052a0, e=(id = 545260288, gen = 209875312)) + 116 at runtime_impl.cc:1252
frame #3: 0x000000010bf37afa tester_io`Realm::Event::has_triggered(this=0x00007ffb98765f5c) const + 106 at event_impl.cc:45
frame #4: 0x000000010bf37eab tester_io`Realm::GenEventImpl::merge_events(wait_for=0x000000012dab59e8) + 667 at event_impl.cc:352
frame #5: 0x000000010bee962a tester_io`void LegionRuntime::LowLevel::CopyRequest::perform_new_dma<2u>(this=0x00007ffb9853f500, src_mem=(id = 1619066880), dst_mem=(id = 1619001344)) + 7210 at lowlevel_dma.cc:3033
frame #6: 0x000000010bed81be tester_io`LegionRuntime::LowLevel::CopyRequest::perform_dma(this=0x00007ffb9853f500) + 1246 at lowlevel_dma.cc:3206
frame #7: 0x000000010bed6a8b tester_io`LegionRuntime::LowLevel::CopyRequest::check_readiness(this=0x00007ffb9853f500, just_check=false, rq=0x0000000000000000) + 2603 at lowlevel_dma.cc:643
frame #8: 0x000000010bee1a06 tester_io`Realm::Domain::copy(this=0x00007ffb9854d2fc, srcs=0x000000012dab7e48, dsts=0x000000012dab7e30, requests=0x000000012dab7ad8, wait_on=(id = 0, gen = 0), redop_id=0, red_fold=false) const + 5910 at lowlevel_d
ma.cc:4915
frame #9: 0x000000010bee028a tester_io`Realm::Domain::copy(this=0x00007ffb9854d2fc, srcs=0x000000012dab7e48, dsts=0x000000012dab7e30, wait_on=(id = 0, gen = 0), redop_id=0, red_fold=false) const + 122 at lowlevel_dma.cc:4815
frame #10: 0x000000010c245ea8 tester_io`LegionRuntime::HighLevel::RegionTreeForest::issue_copy(this=0x00007ffb984069f0, dom=0x00007ffb9854d2fc, op=0x00007ffb98458638, src_fields=0x000000012dab7e48, dst_fields=0x000000012dab7e30, precondition=(i
d = 0, gen = 0)) + 264 at region_tree.cc:3963
frame #11: 0x000000010c245a62 tester_io`LegionRuntime::HighLevel::RegionTreeForest::copy_across(this=0x00007ffb984069f0, op=0x00007ffb98458638, src_ctx=(ctx = 5), dst_ctx=(ctx = 5), src_req=0x00007ffb9844a1b0, dst_req=0x00007ffb98448ff0, src_re
f=0x000000012dab8ac8, dst_ref=0x000000012dab8b80, precondition=(id = 0, gen = 0)) + 3506 at region_tree.cc:2415
frame #12: 0x000000010c06148b tester_io`LegionRuntime::HighLevel::CopyOp::trigger_execution(this=0x00007ffb984585a0) + 12811 at legion_ops.cc:2978
frame #13: 0x000000010c0621ac tester_io`non-virtual thunk to LegionRuntime::HighLevel::CopyOp::trigger_execution(this=0x00007ffb98458638) + 28 at legion_ops.cc:3108
frame #14: 0x000000010c57a80f tester_io`LegionRuntime::HighLevel::Runtime::high_level_runtime_task(args=0x00007ffb98532190, arglen=20, p=(id = 2155872256)) + 911 at runtime.cc:16670
frame #15: 0x000000010bf6671c tester_io`Realm::PreemptableThread::run_task(this=0x00007ffb98406730, task=0x00007ffb98547be0, actual_proc=(id = 2155872256)) + 220 at proc_impl.cc:991
frame #16: 0x000000010bf6754d tester_io`Realm::GreenletTask::run(this=0x00007ffb98511920, arg=0x00007ffb98406730) + 61 at proc_impl.cc:1076
frame #17: 0x000000010bfe6879 tester_io`greenlet::_run(arg=0x00007ffb98406730) + 57 at greenlet-cc.cc:158
frame #18: 0x000000010bfe631e tester_io`_greenlet_start(arg=0x00007ffb98530690) + 110 at greenlet.cc:122
thread #8: tid = 0x6287f, 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #0: 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff90409e0c libsystem_pthread.dylib`_pthread_cond_wait + 693
frame #2: 0x000000010bee44ad tester_io`GASNetCondVar::wait(this=0x00007ffb98600dd0) + 45 at activemsg.h:159
frame #3: 0x000000010bf67bcb tester_io`Realm::GreenletProcessor::execute_task(this=0x00007ffb98600d60) + 411 at proc_impl.cc:1293
frame #4: 0x000000010bf679f7 tester_io`Realm::GreenletThread::thread_main(this=0x00007ffb98601150) + 55 at proc_impl.cc:1102
frame #5: 0x000000010bf66fbc tester_io`Realm::PreemptableThread::thread_entry(data=0x00007ffb98601150) + 156 at proc_impl.cc:1039
frame #6: 0x00007fff90409268 libsystem_pthread.dylib`_pthread_body + 131
frame #7: 0x00007fff904091e5 libsystem_pthread.dylib`_pthread_start + 176
frame #8: 0x00007fff9040741d libsystem_pthread.dylib`thread_start + 13
second proc
thread #1: tid = 0x62868, 0x00007fff8f4a848a libsystem_kernel.dylib`__semwait_signal + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
* frame #0: 0x00007fff8f4a848a libsystem_kernel.dylib`__semwait_signal + 10
frame #1: 0x00007fff9288df5d libsystem_c.dylib`nanosleep + 199
frame #2: 0x00007fff9288ddbd libsystem_c.dylib`sleep + 42
frame #3: 0x000000010b4aa7e4 tester_io`Realm::RuntimeImpl::run(this=0x00007fa52bc052a0, task_id=0, style=ONE_TASK_ONLY, args=0x0000000000000000, arglen=0, background=false) + 3940 at runtime_impl.cc:1103
frame #4: 0x000000010b4a9870 tester_io`Realm::Runtime::run(this=0x00007fff548c5438, task_id=0, style=ONE_TASK_ONLY, args=0x0000000000000000, arglen=0, background=false) + 80 at runtime_impl.cc:138
frame #5: 0x000000010ba56805 tester_io`LegionRuntime::HighLevel::Runtime::start(argc=1, argv=0x00007fff548c5d38, background=false) + 7797 at runtime.cc:15685
frame #6: 0x000000010b4f964a tester_io`LegionRuntime::HighLevel::HighLevelRuntime::start(argc=3, argv=0x00007fff548c5d28, background=false) + 42 at legion.cc:3671
frame #7: 0x000000010b35c390 tester_io`main(argc=3, argv=0x00007fff548c5d28) + 352 at tester_io.cc:244
frame #8: 0x00007fff93c255c9 libdyld.dylib`start + 1
frame #9: 0x00007fff93c255c9 libdyld.dylib`start + 1
thread #2: tid = 0x62874, 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #0: 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff90409e0c libsystem_pthread.dylib`_pthread_cond_wait + 693
frame #2: 0x000000010b39c683 tester_io`IncomingMessageManager::get_messages(this=0x00007fa52bf00860, sender=0x000000010c786ea4, wait=true) + 131 at activemsg.cc:686
frame #3: 0x000000010b39c801 tester_io`Realm::HandlerThread::thread_main(this=0x00007fa52bf00930) + 49 at activemsg.cc:725
frame #4: 0x000000010b448fbc tester_io`Realm::PreemptableThread::thread_entry(data=0x00007fa52bf00930) + 156 at proc_impl.cc:1039
frame #5: 0x00007fff90409268 libsystem_pthread.dylib`_pthread_body + 131
frame #6: 0x00007fff904091e5 libsystem_pthread.dylib`_pthread_start + 176
frame #7: 0x00007fff9040741d libsystem_pthread.dylib`thread_start + 13
thread #3: tid = 0x62876, 0x00007fff8f4a7c22 libsystem_kernel.dylib`__ioctl + 10
frame #0: 0x00007fff8f4a7c22 libsystem_kernel.dylib`__ioctl + 10
frame #1: 0x00007fff8f4a59b7 libsystem_kernel.dylib`ioctl + 159
frame #2: 0x000000010bc5bf2f tester_io`AMUDP_DrainNetwork(amudp_ep*) + 63
frame #3: 0x000000010bc5ab21 tester_io`AM_Poll + 97
frame #4: 0x000000010bbec392 tester_io`gasnetc_AMPoll + 50
frame #5: 0x000000010b39d941 tester_io`do_some_polling() [inlined] gasneti_AMPoll + 5 at gasnet_help.h:597
frame #6: 0x000000010b39d93c tester_io`do_some_polling() [inlined] gasnet_AMPoll at gasnet_help.h:712
frame #7: 0x000000010b39d93c tester_io`do_some_polling() + 28 at activemsg.cc:2084
frame #8: 0x000000010b39dd56 tester_io`gasnet_poll_thread_loop(data=0x0000000000000000) + 38 at activemsg.cc:2100
frame #9: 0x00007fff90409268 libsystem_pthread.dylib`_pthread_body + 131
frame #10: 0x00007fff904091e5 libsystem_pthread.dylib`_pthread_start + 176
frame #11: 0x00007fff9040741d libsystem_pthread.dylib`thread_start + 13
thread #4: tid = 0x62877, 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #0: 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff90409e0c libsystem_pthread.dylib`_pthread_cond_wait + 693
frame #2: 0x000000010b386ace tester_io`LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(this=0x00007fa52bc058f0, dma_thread=0x00007fa52bc059b0, wait_on_empty=true) + 1902 at channel.h:1081
frame #3: 0x000000010b377709 tester_io`LegionRuntime::LowLevel::DMAThread::dma_thread_loop(this=0x00007fa52bc059b0) + 1129 at channel.cc:1448
frame #4: 0x000000010b3d243d tester_io`LegionRuntime::LowLevel::DMAThread::start(arg=0x00007fa52bc059b0) + 29 at channel.h:977
frame #5: 0x00007fff90409268 libsystem_pthread.dylib`_pthread_body + 131
frame #6: 0x00007fff904091e5 libsystem_pthread.dylib`_pthread_start + 176
frame #7: 0x00007fff9040741d libsystem_pthread.dylib`thread_start + 13
thread #5: tid = 0x62878, 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #0: 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff90409e0c libsystem_pthread.dylib`_pthread_cond_wait + 693
frame #2: 0x000000010b386ace tester_io`LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(this=0x00007fa52bc058f0, dma_thread=0x00007fa52bc05dd0, wait_on_empty=true) + 1902 at channel.h:1081
frame #3: 0x000000010b377709 tester_io`LegionRuntime::LowLevel::DMAThread::dma_thread_loop(this=0x00007fa52bc05dd0) + 1129 at channel.cc:1448
frame #4: 0x000000010b3d243d tester_io`LegionRuntime::LowLevel::DMAThread::start(arg=0x00007fa52bc05dd0) + 29 at channel.h:977
frame #5: 0x00007fff90409268 libsystem_pthread.dylib`_pthread_body + 131
frame #6: 0x00007fff904091e5 libsystem_pthread.dylib`_pthread_start + 176
frame #7: 0x00007fff9040741d libsystem_pthread.dylib`thread_start + 13
thread #6: tid = 0x6287a, 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #0: 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff90409e0c libsystem_pthread.dylib`_pthread_cond_wait + 693
frame #2: 0x000000010b386ace tester_io`LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(this=0x00007fa52bc058f0, dma_thread=0x00007fa52bc06200, wait_on_empty=true) + 1902 at channel.h:1081
frame #3: 0x000000010b377709 tester_io`LegionRuntime::LowLevel::DMAThread::dma_thread_loop(this=0x00007fa52bc06200) + 1129 at channel.cc:1448
frame #4: 0x000000010b3d243d tester_io`LegionRuntime::LowLevel::DMAThread::start(arg=0x00007fa52bc06200) + 29 at channel.h:977
frame #5: 0x00007fff90409268 libsystem_pthread.dylib`_pthread_body + 131
frame #6: 0x00007fff904091e5 libsystem_pthread.dylib`_pthread_start + 176
frame #7: 0x00007fff9040741d libsystem_pthread.dylib`thread_start + 13
thread #7: tid = 0x6287d, 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #0: 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff90409e0c libsystem_pthread.dylib`_pthread_cond_wait + 693
frame #2: 0x000000010b3c64ad tester_io`GASNetCondVar::wait(this=0x00007fa52bf00af0) + 45 at activemsg.h:159
frame #3: 0x000000010b449bcb tester_io`Realm::GreenletProcessor::execute_task(this=0x00007fa52bf00a80) + 411 at proc_impl.cc:1293
frame #4: 0x000000010b4499f7 tester_io`Realm::GreenletThread::thread_main(this=0x00007fa52bc06a50) + 55 at proc_impl.cc:1102
frame #5: 0x000000010b448fbc tester_io`Realm::PreemptableThread::thread_entry(data=0x00007fa52bc06a50) + 156 at proc_impl.cc:1039
frame #6: 0x00007fff90409268 libsystem_pthread.dylib`_pthread_body + 131
frame #7: 0x00007fff904091e5 libsystem_pthread.dylib`_pthread_start + 176
frame #8: 0x00007fff9040741d libsystem_pthread.dylib`thread_start + 13
thread #8: tid = 0x62880, 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #0: 0x00007fff8f4a8136 libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff90409e0c libsystem_pthread.dylib`_pthread_cond_wait + 693
frame #2: 0x000000010b3c64ad tester_io`GASNetCondVar::wait(this=0x00007fa52bf00c50) + 45 at activemsg.h:159
frame #3: 0x000000010b449bcb tester_io`Realm::GreenletProcessor::execute_task(this=0x00007fa52bf00be0) + 411 at proc_impl.cc:1293
frame #4: 0x000000010b4499f7 tester_io`Realm::GreenletThread::thread_main(this=0x00007fa52bf00fc0) + 55 at proc_impl.cc:1102
frame #5: 0x000000010b448fbc tester_io`Realm::PreemptableThread::thread_entry(data=0x00007fa52bf00fc0) + 156 at proc_impl.cc:1039
frame #6: 0x00007fff90409268 libsystem_pthread.dylib`_pthread_body + 131
frame #7: 0x00007fff904091e5 libsystem_pthread.dylib`_pthread_start + 176
frame #8: 0x00007fff9040741d libsystem_pthread.dylib`thread_start + 13
from legion.
streichler
commented on July 17, 2024
Ok, so this line is really fishy:
frame #2: 0x000000010bfc9da4 tester_io`Realm::RuntimeImpl::get_event_impl(this=0x00007ffb984052a0, e=(id = 545260288, gen = 209875312)) + 116 at runtime_impl.cc:1252
Was this test running long enough to have gone through over 200M generations of this event? If not, we need to see where the generation number got corrupted.
from legion.
dotnwat
commented on July 17, 2024
I don't have an intuition for the rate of generation id creation, but I suspect that is a bad value as the entire test typically lasts less than about 10 seconds. I'll dig into the invalid generation values.
from legion.
streichler
commented on July 17, 2024
Noah, any update on this?
from legion.
dotnwat
commented on July 17, 2024
Unfortunately I don't, but I haven't seen this occur recently. There was a whole bunch of issues that we ran into when we were running Legion on some very old and slow hardware (presumably this is effective at increasing the size of windows that race conditions can trigger). But we've been busy getting stuff running for performance.
from legion.
streichler
commented on July 17, 2024
Ok, I'm going to close this for now then. Feel free to reopen it (or just open a new issue) if the problem comes back.
from legion.
Related Issues (20)
- weirdness around NUMA memory allocation on different machines HOT 8
- Realm: HSA_STATUS_ERROR_OUT_OF_RESOURCES on Crusher HOT 7
- hang on 4 nodes of summit HOT 5
- RFC: Proposal for Relaxed Coherence Semantics HOT 4
- Realm: GASNetEx backend very slow / hang during sysmem registration HOT 1
- Regent: __demand(__index_launch) ast.typed.expr.Cast has no field 'value' (in lookup) HOT 6
- Trying DG-Legion with collective HOT 61
- Regent: conflict between RDIR and index launches with multiple identical parameters
- Freeze in predicated task launch HOT 14
- Profiler: Associate point tasks with their producing index launch
- Zero-Copy Memory Query should not return empty via best_affinity_to HOT 4
- Assertion `current >= cnt' failed HOT 7
- Regent: Non-deterministic issue at compile time HOT 3
- Regent: CUDA generator fails with particular triple nested loops HOT 6
- Regent: Cannot format ptr
- Unable to run legion implementation HOT 25
- Regent: lifting variables to Futures across `__local` tasks boundaries
- Assertion error from recent CR HOT 3
- CMake: Compilation error of legion_redop.cu with -std=c++17 HOT 9
- Regent: Future leak in Pennant
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from legion.