Comments (4)
That might be caused by a known issue with yet unknown solution: Kubernetes sometimes “loses” the connection without closing it. Since the connection is technically opened, Kopf does not reconnect and believes that nothing happens in the cluster.
If you search through issues, Azure is mentioned several times as especially affected by this. My guess is that the problem is in load balancers and their connections to the real control plane (in the chain: kopf->lb->k8s). Unconfirmed though.
Often, setting the client-side (i.e. kopf-side) connection timeout helps (see settings). Not the best solution, but it works: the operator might be not noticing the changes for the configured time out (e.g 10 mins), or will have to reconnect too often (if you set it to 1m). The “good” value depends on your individual case, there is no “good default”.
I see no way to fix this on the Kopf side, unless there is some kind of ping-pong machinery in k8s above low-level tcp.
from kopf.
Hi, thanks for the quick response.
I'm not quite sure how I'm going to proceed yet, but I'll try to look into something more related to Azure and analyze the traffic further.
I will also validate changes to the connection timeout.
Thanks
from kopf.
We are facing similar problem with Azure too. I was thinking about configuring a liveness probe which will cause a request to k8s api somehow to keep the connection alive, in case for whatever reason the LB kills the connection after some time if they are not used. @nolar what do you think?
from kopf.
@francescotimperi The key problem is the existing connection, not a new one. The probe will be showing success since new tcp connections are landing fine. It is the existing connection that remains connected but dysfunctional. You need a response from k8s there to validate it — this is what I meant by ping-ponging. But I saw no such feature in k8s.
from kopf.
Related Issues (20)
- Flakey behavior of on.create handler not reacting to CRO creation event. HOT 1
- Automating Kopf Operator Code Updates: Seeking a Convenient Solution to Avoid Tedious Run Commands HOT 1
- Controlling Kopf Operator Behavior to Prevent Unwanted Object Creation HOT 8
- Using named import in kopf
- Handle large resource spec being annotated to `last-handled-configuration` HOT 3
- Namespace deletion is stuck when using namespace selector startup mode HOT 1
- Infinite watch-streams stopping immediately with no obvious reasons why. The watched resources do then spin up.
- Liveness probe stops working HOT 2
- Kubernetes client not configured in startup handler HOT 3
- Admission Controller Path
- `FieldSpec` and `resolve` does not support list access
- Attach to container start event
- Behaviour of timer with finalizer
- Finalisers race condition
- Finalizers from other controllers conflicting with kopf finalizer
- Kopf does not restart after 429 too many requests error HOT 1
- How to avoid Errors accessing metrics API and CRDs HOT 1
- add cwd to sys.path HOT 4
- Allow kopf.adopt to Communicate Resource Readiness to ArgoCD HOT 4
- Object stuck in loop with inconsistent status updates and handler failure HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kopf.