Comments (10)
nosmicek
commented on August 24, 2024
Solved by downgrading helm chart from 1.39.1 to 1.37.0 (haproxy ingress controller 1.11.2->1.10.11) with exactly the same configuration.
After downgrade we don't see any exhibit of this erroneous behaviour, clearly this is some new problem introduced in 1.11 version of the controller. With it, its practically unusable in dynamic environments where pods scale up/down often.
from kubernetes-ingress.
ivanmatmati
commented on August 24, 2024
Hi @nosmicek , Thanks for reporting, I'll check what happened.
from kubernetes-ingress.
nosmicek
commented on August 24, 2024
Hi, thanks, if you need some info about our setup or configuration etc. don't hesitate to ask, I have some limited time dedicated to solving this.
Also, another issue that I've noticed when working on it was, that when I configured hard-stop-after and close-spread-time to 300000 (5minutes), the haproxy ingress controller ended up periodically reloading every 10minutes on 1.11.2 version regardless of our service scaling (was constant for the whole period of test). Again due to this prometheus endpoint. We tried this as a workaround to reloads, but it ended up in much worse scenario.
from kubernetes-ingress.
ivanmatmati
commented on August 24, 2024
Hi @nosmicek , can you try with the nightly build ? There's a change that could solve the issue. To use it just replace the tag in your yaml. Switch from latest to nightly.
from kubernetes-ingress.
nosmicek
commented on August 24, 2024
Hi @ivanmatmati, with the latest, the issue is still present. I tried it today.
from kubernetes-ingress.
ivanmatmati
commented on August 24, 2024
Latest or nightly ?
from kubernetes-ingress.
nosmicek
commented on August 24, 2024
I tried now with nightly and the problem is even worse, I've got lots of reloads due to prometheus even when not scaling our backend service pods. It's really hard to distinguish whether it is caused by scaling or not, but it seems that it happens when scaling.
from kubernetes-ingress.
ivanmatmati
commented on August 24, 2024
I don't see these reloads due to prometheus. Can you check the commit you're on ? On my side it's c3cd22c.
from kubernetes-ingress.
TubbyStubby
commented on August 24, 2024
The same happened with me and I was on chart version 1.39.4. It got resolved after downgrading to 1.37.0 as suggested by @nosmicek
this was happening whenever pods were being added or removed by autoscaler. I got these logs. Also noticed that the status was showing nocheck in stats for all backends which got fixed after downgrade without any config change.
[NOTICE] (153) : haproxy version is 2.8.9-1842fd0
[WARNING] (153) : Former worker (8354) exited with code 0 (Exit)
[WARNING] (8375) : Server ada_switch-service_wshttp/SRV_1 is UP/READY (leaving forced maintenance).
[WARNING] (8392) : Server ada_switch-service_wshttp/SRV_1 is UP/READY (leaving forced maintenance).
[WARNING] (153) : Former worker (8350) exited with code 0 (Exit)
[WARNING] (8392) : Server ada_switch-service_wshttp/SRV_2 is going DOWN for maintenance. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] (8375) : Server ada_switch-service_wshttp/SRV_2 is going DOWN for maintenance. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2024/05/30 07:08:03 INFO service/service.go:157 [transactionID=5e49ea04-738f-4723-9ead-08c52417cf12] reload required : Service 'ada/switch-service': backend 'ada_switch-service_wshttp' updated: map[HTTPConnectionMode:[ http-server-close]]
2024/05/30 07:08:04 INFO controller/controller.go:196 [transactionID=5e49ea04-738f-4723-9ead-08c52417cf12] HAProxy reloaded
[NOTICE] (153) : Reloading HAProxy
[WARNING] (8392) : Proxy healthz stopped (cumulated conns: FE: 86, BE: 0).
[WARNING] (8392) : Proxy http stopped (cumulated conns: FE: 5, BE: 0).
[WARNING] (8392) : Proxy https stopped (cumulated conns: FE: 233, BE: 0).
[WARNING] (8392) : Proxy stats stopped (cumulated conns: FE: 3, BE: 0).
[WARNING] (8392) : Proxy ada_switch-service_wshttp stopped (cumulated conns: FE: 0, BE: 5).
[WARNING] (8392) : Proxy haproxy-controller_default-local-service_http stopped (cumulated conns: FE: 0, BE: 0).
[WARNING] (8392) : Proxy haproxy-controller_prometheus_http stopped (cumulated conns: FE: 0, BE: 0).
[NOTICE] (153) : New worker (8413) forked
[NOTICE] (153) : Loading success.
[NOTICE] (153) : haproxy version is 2.8.9-1842fd0
[WARNING] (153) : Former worker (8392) exited with code 0 (Exit)
2024/05/30 07:08:06 INFO service/service.go:157 [transactionID=3e1b981d-33a0-4672-89a9-aaae0888008b] reload required : Service 'ada/switch-service': backend 'ada_switch-service_wshttp' updated: map[HTTPConnectionMode:[ http-server-close]]
2024/05/30 07:08:06 INFO controller/controller.go:196 [transactionID=3e1b981d-33a0-4672-89a9-aaae0888008b] HAProxy reloaded
[NOTICE] (153) : Reloading HAProxy
[WARNING] (8375) : Proxy healthz stopped (cumulated conns: FE: 38, BE: 0).
[WARNING] (8375) : Proxy http stopped (cumulated conns: FE: 1, BE: 0).
[WARNING] (8375) : Proxy https stopped (cumulated conns: FE: 101, BE: 0).
[WARNING] (8375) : Proxy stats stopped (cumulated conns: FE: 2, BE: 0).
[WARNING] (8375) : Proxy ada_switch-service_wshttp stopped (cumulated conns: FE: 0, BE: 1).
[WARNING] (8375) : Proxy haproxy-controller_default-local-service_http stopped (cumulated conns: FE: 0, BE: 0).
[WARNING] (8375) : Proxy haproxy-controller_prometheus_http stopped (cumulated conns: FE: 0, BE: 0).
[NOTICE] (153) : New worker (8396) forked
[NOTICE] (153) : Loading success.
from kubernetes-ingress.
stale
commented on August 24, 2024
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
from kubernetes-ingress.
Related Issues (20)
- http -> https default redirect not working by default HOT 13
- Deployment yaml still point to privileged port after s6 upgrade HOT 3
- frontend tcp-514: cannot bind socket (Permission denied) for [0.0.0.0:514] HOT 2
- When specifying a server-proto annotation the check parameter is omitted
- Using both annotations "haproxy.org/cr-backend" and "haproxy.org/<parameter>" HOT 3
- Docs: when adjusting --sync-period, must also adjust initialDelaySeconds HOT 4
- Controller restarts haproxy every 5 minutes with message "restart required : Global config updated: " HOT 4
- Docs: Add info about --job-check-crd to documentation HOT 1
- Ingress Pods Fail to Start HOT 5
- Try to understand the mode for tcpcr in 3.0.0 HOT 4
- how enable only high secure ssl ciphers in haproxy ingress? HOT 3
- change route-acl into an array so multiple ACL's can be defined here. HOT 1
- Named backend server instead of generic SRV_x name HOT 6
- Not able to setup retry-on functionality globally, I tried multiple values.yaml ! Please correct me if i am wrong ! HOT 2
- Some clarification needed for handling canary deployments HOT 3
- Runtime API changes not getting synced between haproxy replicas HOT 3
- Unable to Configure Multiple Backends on the Same Port in TCP CRD HOT 5
- Possibility to add custom stick tables HOT 2
- rate-limit-requests
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kubernetes-ingress.