Stuck in EAGLE 50 about swift HOT 14 CLOSED

Hi, what MPI version and fabric are you using? We have seen some implementations not behaving correctly.

from swift.

I'm using openmpi 3.1.3 with InfiniBand.
Thanks for the quick response.

from swift.

Ok. So that seems fine. And what version of the OFED driver are you using? The regular Linux-kernel one or the Mellanox-optimised?

from swift.

Also, what transport library are you using in OpenMPI?

We recommend psm and not psm2. That is running with --mca btl vader,self --mca mtl psm.

from swift.

Ok. So that seems fine. And what version of the OFED driver are you using? The regular Linux-kernel one or the Mellanox-optimised?

I think the Mellanox-optimised version

We recommend psm and not psm2. That is running with --mca btl vader,self --mca mtl psm.

I will try that

from swift.

 Ok. So that seems fine. And what version of the OFED driver are you using? The regular Linux-kernel one or the Mellanox-optimised?

I think the Mellanox-optimised version

Right, then that is likely the issue. Their curent driver hangs if too many asynchronous communications are in-flight at a given point in time. SWIFT makes extensive use of this mechanism so you may be facing this issue here.

from swift.

Would removing the Mellanox driver fix the issue?

from swift.

I can only speculate as I have never seen this issue on machines where we have control over things but it may help.

Otherwise, trying a different mtl in OpenMPI might help (instead of changing driver).

from swift.

Right, then that is likely the issue. Their curent driver hangs if too many asynchronous communications are in-flight at a given point in time. SWIFT makes extensive use of this mechanism so you may be facing this issue here.

I seem to get the same problem on a system with the regular driver.

from swift.

Did you try changing the mtl to psm?

from swift.

Hi Jan,

Have you had some luck with the code?

from swift.

I'm having some trouble as the OpenMPI version I'm using apparently doesn't support psm and installing it with psm is a little problematic at the moment, but I'm still on it.

from swift.

I'm also using Slurm as job scheduler, I forgot to mention that earlier. I hope that that is not interfering with anything.

from swift.

Did you try changing the mtl to psm?

using psm did not seem to resolve the issue.

Edit: I made another mistake while running. Using psm does seem to resolve the problem

from swift.