Long worker-shutdown-timeout causes long-lived connections to fail about ingress-nginx HOT 3 OPEN

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

from ingress-nginx.

I'm also happy to share some extra evidence privately. But I was able to:

• use prometheus to pull a list of IPs for pods before and after the deployment rollout
• query ingress-nginx logs for requests to the service in question, filtering for IPs in the first list, but omitting requests to ips in the second list

The log volume shows a massive drop off in requests right as the rollout happens (which I would expect). However, a small tail of requests continues to the old pods, despite the fact that they don't exist (or at least aren't used by the app in question anymore)

from ingress-nginx.

/remove-kind bug
/triage needs-information

I think that the user her #11508 also reports the same behavior..

While we wait for other comments and maybe some expert opinion, I made an assumption that in a multi-replica use-case, the routing and loadbalancing of traffic will be the same regardless of the backend-protocol from controller to backend being gRPC or HTTP. Let me know if my assumption is wrong. It will take me too long to dive into the code and ascertain this.

So I tested with a 5 replica deployment using image nginx:alpine and generating at least 1 new connection per second with multiple sessions, to that deployment's HTTP backend-protocol ingress.

I could not reproduce the problem indicated in these 2 issues. I did see a transition related response code though.

So I am inclined to think that off-beat connection requesting client needs to handle the transitions better. It seems like such client have a myopic view that neither they use affinity/persistence nor do they accommodate the transition related events like graceful-drain of connections and route to another backend pod.

In any case, it will help a lot to get a step-by-step guide to reproduce the problem.

from ingress-nginx.