Unresponsive leader and lost DOWNs about gproc HOT 8 CLOSED

It's certainly not a feature! :)

I'm looking into it.

from gproc.

I have some ideas, but the trickiest part of the problem is that a netsplit occurs. Only a few of the gen_leader versions (e.g. garret-smith/gen_leader_revival) have some support for netsplits, and at least when I try this scenario with garret-smith's version, it doesn't seem to do the right thing.

However, a few things come to mind:

• Gen_leader is further delayed after the node ping times out, unless -kernel dist_auto_connect is set to once or never. The reason is that each message to the unresponsive node will lead to a connection attempt, which will then hang for a while.
• Until we have solid netsplit handling in both gen_leader and gproc, I recommend that you set up your own higher-level supervision. One way to do this is to set -kernel dist_auto_connect once, as mentioned above, then have a process on each node that periodically sends a UDP message to the other known (but not necessarily connected) nodes. If you receive a UDP message from a node that's not in the nodes() list, you have a netsplit situation. If you have no better strategy available, you can then restart the nodes that make up one of the 'islands'.

from gproc.

@msadkov There exists an application to detect network splits for mnesia and hibari. It would need some customisation for gproc. Nevertheless, it might be of help to you.

The application is here => https://github.com/hibari/partition-detector

The admin documentation is here => http://hibari.github.com/hibari-doc/hibari-sysadmin-guide.en.html#partition-detector

from gproc.

@uwiger @norton thank you for your replies! I'm aware of gen_leader/gproc not being able to handle net splits, so -kernel dist_auto_connect was set to once in this case (I should have mentioned this in my first post, sorry).. with that said, this situation doesn't look like a net split, but rather node going down (not immediately, but after a timeout), right? and after unresponsive node disappears from nodes list (which means a terminated connection, AFAIU) there is no timeout involved anymore, since I can get an immediate DOWN message after calling erlang:monitor with a dead pid sitting in gproc's ets..

from gproc.

I'm making som progress getting gproc to heal after netsplits, as well as doing proper monitoring. I don't have a solution for handling conflicts yet, and need to fix some regression bugs. I'll keep you informed..

from gproc.

Thank you!

from gproc.

Going through the issues list, closing out issues. This one is not yet resolved, so I'll leave it open. Sorry about the delay.

from gproc.

Closing this issue. Feel free to try out the locks_leader branch which should handle netsplits more robustly.

from gproc.