Faced the following problem when using grpc-lb-consul
:
- gRPC service runs on some HOST:PORT
- then it exits
- another (HTTP/1) service starts on same HOST:PORT (really rare case, but it happened with me)
- balancer continues to query that HOST:PORT (with new HTTP/1 service running) and logs many errors:
error retrieving load score for EXITED_GRPC_SERVICE from HOST:PORT: rpc error: code = 13 desc = transport is closing
It's logged here: balancer/backend.go#L79
New HTTP/1 service receives HTTP/2 PRI method requests (which it cannot process; but even if it could - another error would be returned - like "no such endpoint" or similar).
My suggestion is to close backend on errors (probably, only on some specific errors - like "closing" or similar) and exclude it from a set of backends for service.
Backends are re-discovered (so closed/connected) each balancer.Config.Discovery.Interval
(defaults to 5m).
So even if there is no new service, running on that HOST:PORT, grpc-lb-consul
may flood the logs with some other errors (like "couldn't connect" or so).
So "self-closing" backends would be nice to have.
Also, probably, health/load checks would be better done with gRPC streams? Not that it's very important, but if another gRPC service "reuses" address, it may be reported to "previous" service client while services are not re-discovered. But this is rare case, so, probably, not worth bothering.