segment fault in activemsg about legion HOT 7 CLOSED

Zhihao, can you provide the exact command line you're using?

from legion.

I think the failure could be reproduced by following command line:

GASNETBACKTRACE=1 mpirun -n 2 --bind-to none -H n0000,n0001 ./fluid3d
-input in_300K.fluid -nbx 2

in which case, we try to partition the entire computation into 2 pieces,
and run each piece on one node.

Sincerely,
Zhihao

On Thu, Sep 3, 2015 at 1:11 PM, streichler [email protected] wrote:

Zhihao, can you provide the exact command line you're using?

—
Reply to this email directly or view it on GitHub
#50 (comment)
.

from legion.

My bad...please ignore the previous email, which leads to a fluid3d failure
case. I found the active message failure case when I run dma_random test in
dma branch. I am not sure if we can reproduce it in master branch.

On Thursday, September 3, 2015, Zhihao Jia [email protected] wrote:

I think the failure could be reproduced by following command line:

GASNETBACKTRACE=1 mpirun -n 2 --bind-to none -H n0000,n0001 ./fluid3d
-input in_300K.fluid -nbx 2

in which case, we try to partition the entire computation into 2 pieces,
and run each piece on one node.

Sincerely,
Zhihao

On Thu, Sep 3, 2015 at 1:11 PM, streichler <[email protected]
javascript:_e(%7B%7D,'cvml','[email protected]');> wrote:

Zhihao, can you provide the exact command line you're using?

—
Reply to this email directly or view it on GitHub
#50 (comment)
.

Sincerely,
Zhihao

from legion.

The following steps could help reproduce #50.

Step 1: check out to recent dma branch
Step 2: enter test/dma_random folder
Step 3: GASNET_BACKTRACE=1 mpirun -n 2 --bind-to none -H n0000,n0001
dma_random -ll:rsize 1024 -ll:dsize 1024 -ll:ahandlers 2

Sincerely,
Zhihao

On Thu, Sep 3, 2015 at 1:31 PM, Zhihao Jia [email protected] wrote:

My bad...please ignore the previous email, which leads to a fluid3d
failure case. I found the active message failure case when I run dma_random
test in dma branch. I am not sure if we can reproduce it in master branch.

On Thursday, September 3, 2015, Zhihao Jia [email protected] wrote:

I think the failure could be reproduced by following command line:

GASNETBACKTRACE=1 mpirun -n 2 --bind-to none -H n0000,n0001 ./fluid3d
-input in_300K.fluid -nbx 2

in which case, we try to partition the entire computation into 2 pieces,
and run each piece on one node.

Sincerely,
Zhihao

On Thu, Sep 3, 2015 at 1:11 PM, streichler [email protected]
wrote:

Zhihao, can you provide the exact command line you're using?

—
Reply to this email directly or view it on GitHub
#50 (comment)
.

Sincerely,
Zhihao

from legion.

Zhihao, when I run the above command now, I reliably get this error instead:
dma_random: ../../runtime/realm/mem_impl.cc:604: virtual void Realm::RemoteMemory::get_bytes(off_t, void*, size_t): Assertion kind == MemoryImpl::MKIND_RDMA' failed.
*** Caught a fatal signal: SIGABRT(6) on node 0/2`

Can you see if this is a new issue hiding the old one, or maybe the old one is fixed and this one
was hiding behind it?

from legion.

Hi Sean,

The bug you encountered is a issue happening before #50, which is caused by
check_correctness task trying to read a remote instance (not RDMA-able). I
have pushed a fix into dma branch. You should be able to reproduce #50 on
n0000,n0001. By the way, don't forget to remove disk_file* before every
execution, otherwise, disk memory sanity check may fail.

I am working on merging your threading changes into dma branch. Do you
think if the threading changes may fix #50?

Sincerely,
Zhihao

On Wed, Sep 9, 2015 at 9:23 AM, streichler [email protected] wrote:

Zhihao, when I run the above command now, I reliably get this error
instead:
dma_random: ../../runtime/realm/mem_impl.cc:604: virtual void
Realm::RemoteMemory::get_bytes(off_t, void_, size_t): Assertion `kind ==
MemoryImpl::MKIND_RDMA' failed.
*_* Caught a fatal signal: SIGABRT(6) on node 0/2

Can you see if this is a new issue hiding the old one, or maybe the old
one is fixed and this one
was hiding behind it?

—
Reply to this email directly or view it on GitHub
#50 (comment)
.

from legion.

This has been replaced by #61, which appears to be an issue inside the new DMA code.

from legion.