Problem in IOF subsystem about prrte HOT 5 CLOSED

@rhc54 sure, I'll take a crack at it !

how can you reproduce this issue ?
does a simple non PMIx program that writes a lot to stdout can do the trick ?

from prrte.

You can find my "cycle.sh" here: https://github.com/pmix/pmix-tests/blob/master/prrte/cycle.sh

from prrte.

I think I've thought through the issue, though I haven't attempted a fix. I believe the problem is a race condition between PRRTE sending a job-terminated notice to prun and the IOF sending the output to prun. If the job-terminated notification gets delivered first, then prun will exit - yet the IOF doeesn't know that has happened and still attempts to write the apps output down the (now defunct) socket.

The issue is that prun is not a child of PRRTE - it's just a connected tool. We need to ensure that any IOF collected from the associated app gets sent to prun before we send the termination notification. We then need to ensure that prun outputs any received IO prior to actually terminating - guess we need an "IOF flush" somewhere in prun's finalization routine.

HTH

from prrte.

@ggouaillardet I spent the day working on this and found that my above premise was incorrect - this had nothing really to do with the IOF. The problem appears to be in either the notification or the messaging system of the client.

What I found was that we were caching notifications for a long period of time, as you previously noted. However, since PRRTE wasn't providing set targets for those job-termination notices, every prun would get a flood of notices about prior jobs when it connected to the server. The client event system would filter those so that prun itself never saw them, but once you built up a lot of them, prun would start to hang under the flood.

Adding the IOF into the mix seemed to make it happen faster, though I couldn't solidly confirm that impression. I'm inclined to believe it is solidly in the event system because I was able to fix the problem by revamping solely the event system. However, the revamp also eliminated the message flood, and so I might have solved only the symptom. We need to stress the server-to-client message system to be sure.

My fixes are in two PRs, one for PMIx itself and the other for PRRTE:
openpmix/openpmix#1003
#175

With those fixes, I am able to run my cycle.sh test essentially forever (well, I tested 10k iterations without a problem, so that's forever to me!).

Would you have a chance to check out the messaging system? I'd like to ensure my fixes aren't just masking the root cause of the problem.

from prrte.

Looks like this is fixed

from prrte.