Propose a client side failover strategy about armeria HOT 5 CLOSED

@shelocks wrote:

what do you think about these candidates

(1, 2) and (3) are not mutually exclusive. In case of network issues, you can always retry for all proposed ideas.

the armeria is async,maybe catching the network exception in this way is not easy(need some sync test?) and has performance impact.

Not really, you can just add a listener to the Future returned by delegate().invoke(...) and decide to retry there:

Promise<T> promise = eventLoop.newPromise();
delegate().invoke(...).addListener(f -> {
    ...
    if (success or non-network failure) {
        promise.setSuccess/Failure(....);
    } else {
        retry();
    }
});
return promise;

from armeria.

@shelocks Also, (1) and (2) are not mutually exclusive either. A user might want to use both of them.

from armeria.

In general I'm not a fan of automatic retry because there's pretty much no way to know how far your request got when it failed, even due to ClosedSocketException or whatever. Pretty much the only network failure I'd be comfortable with retrying is if the inital bits never got sent over the wire in the first place - is this something we can distinguish (seems difficult in HTTP2)? Retrying in such a situation seems great if it's possible.

Also, DB stuck, full GC stuck, etc usually need to all be handled anyways, so handling network failure in the RPC layer but leaving other random failures up to the user seems not so useful.

It's of course ok to provide a RetryingClient in case users do have a use for 3), but IMO it's really only useful for read-only APIs, which limits its applicability a lot.

I'm not sure what the difference is between 2) and 3). Both seem to rely on request failure, which have the same issues of not knowing whether it's safe to retry. Does 2 imply not retrying but using another endpoint on the next request? In that case, 2) seems good too, but not enough by itself.

1. is pretty much the only way I've seen to reliably do RPC. It has the problem with random network failures that you mention, but as I mentioned above, apps need to be able to handle random failure regardless. But it will allow for safe rolling restarts - armeria server already has support for graceful shutdown using the health endpoint. A simple implementation would poll the health path on all backends, a more complex (but scalable) implementation would poll an etcd or zookeeper file. And it shouldn't be too hard to incorporate 2) into this by treating request failures by some pattern as unhealthy.

from armeria.

Agreed. Automatic retrial is only applicable in a very specific situation.

from armeria.

I guess this is pretty much solved by the RetryingClient written by @minwoox. Great to see your effort reaches the goal. 👍

from armeria.