PMIX clients time out on start-up using PRRTE as launcher about prrte HOT 14 CLOSED

@tonycurtis I believe this is fixed but don't have a ready way to test it - can you please give the head of PRRTE master a try and let me know (close this issue if it works)?

from prrte.

from prrte.

Thanks!

from prrte.

Still happening on bigger runs (wasn't able to get >4 nodes earlier). >7 nodes seems be the trigger point right now. Same program running fine with OpenMPI as launcher.

--------------------------------------------------------------------------
ORTE has lost communication with a remote daemon.

  HNP daemon   : [[34392,0],0] on node cn097
  Remote daemon: [[34392,0],5] on node cn010

This is usually due to either a failure of the TCP network
connection to the node, or possibly an internal failure of
the daemon itself. We cannot recover from this failure, and
therefore will terminate the job.
--------------------------------------------------------------------------

from prrte.

from prrte.

Well, actually, we're apparently switching over on this cluster fully to SLURM next Monday and it's the only PBS/Torque system I have access to at the moment, so I won't be able to test after that.

from prrte.

(we should probably re-open this issue to keep track of it)

@tonycurtis Can you try with Open MPI v3.1.4 on that machine?
Open MPI hit a similar issue with its launcher in the v3 series that does not exist in the v4 or master series. I wonder if it might be a similar root cause (maybe a similar eventual fix).

• open-mpi/ompi#7087 (comment)

from prrte.

Open-MPI 3.1.4 & 4.0.2 worked with 8 nodes

from prrte.

Launch hanging with Open-MPI github master @ ecd990a67cbc5ce85328202858c90aae4d7fe122 (any number of processes) & PMIx master. No timeout message so far, just sits there...

from prrte.

I don't seem to have the proper rights to re-open this issue. But we should do so so we don't lose track of this.

from prrte.

Just managed to test with new SLURM setup: same error with 8 nodes

from prrte.

Looked into this again (we now have SLURM everywhere).

prted is leaking memory with >= 8 nodes until it triggers the oom-killer on all the non-launch nodes. And it's the number of nodes that is important here, not the proc count per node.

from prrte.

Looks like fixed

from prrte.

Woohoo! Thanks!

from prrte.