Node Healing Strategy

1. a strategy to figure out when a node is down. the one we have now is not appropriate.
2. an implementation of that strategy that doesn't require locks
3. a strategy to handle all connections on that node if the node goes down. This can have locks, because it's rare.
4. a strategy to re-add the node to the pool, without locks,when the node comes online.
5. a strategy to know when a node is back online.

Problems

Nodes are not being killed off from on the fly connections. After a node is closed, a call should be made to DeactivateNode - verify this for on the fly connections. So we need to know when to take a node out, but the "putting it back in" is what we need to fix.

@OJ has located these sections of the codebase and made them out to be significant:

https://github.com/DistributedNonsense/CorrugatedIron/blob/master/CorrugatedIron/Comms/LoadBalancing/RoundRobinStrategy.cs#L62
https://github.com/DistributedNonsense/CorrugatedIron/blob/master/CorrugatedIron/RiakCluster.cs#L158
https://github.com/DistributedNonsense/CorrugatedIron/blob/master/CorrugatedIron/RiakCluster.cs#L165
https://github.com/DistributedNonsense/CorrugatedIron/blob/master/CorrugatedIron/RiakCluster.cs#L146

from riak-dotnet-client.

I think the detection of exceptions causing sockets to close should be handled in RiakConnection. The idle timeout should be on RiakPbcSocket.

See:
https://github.com/taliesins/CorrugatedIron/blob/master/CorrugatedIron/Comms/RiakConnection.cs
https://github.com/taliesins/CorrugatedIron/blob/master/CorrugatedIron/Comms/RiakPbcSocket.cs

from riak-dotnet-client.

As with #153 and #116, please send a PR to make review easier.

from riak-dotnet-client.

I am afraid I have deviated tremendously from the original branch as I have made my version fully async and reactive. So its big changes.

So its going to be an act of cherry picking which bits you want.

I will manually port them over and send a pull request, as soon as I get a gap :>

from riak-dotnet-client.