Goroutine leak in `Instance.SetPeers` about gubernator HOT 3 CLOSED

I couldn't reliably build graceful shutdowns of the connection using the WaitForStateChange API, so ended up using a sync.WaitGroup to track in-flight requests instead.

The changes are in my PR #28.

from gubernator.

This would be great! As you mentioned, we are not seeing this in our prod environments because our peer sets are pretty stable. The hard part of this would be knowing when all in flight requests to a peer have completed. It's possible that we could just assume the peer is no longer fielding requests because it's been removed from the peer set. Thoughts?

from gubernator.

It appears the there's an experimental API to wait for a given GRPC connection to enter a state, which could be used to build graceful shutdowns (https://godoc.org/google.golang.org/grpc#ClientConn.WaitForStateChange). A shutdown process that accounts for this may be as follows:

1. Close the request queue
2. Send the pending batch
3. Wait for connection state to be idle
4. Close the connection

I'll investigate whether this approach is viable.

from gubernator.