Comments (6)
It looks like it's related to a bug in Kubernetes version 1.27.
Setting a mirror pod's phase to Succeeded or Failed can prevent the corresponding static pod from restarting due mutation of a Kubelet cache. (https://github.com/kubernetes/kubernetes/pull/116482, [@smarterclayton](https://github.com/smarterclayton)) [SIG Node]
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.28.md
After upgrading to 1.28 we have not seen this issue resurface.
from charts.
Did you check if some event shows in the pod that the status was "succeeded"?
from charts.
I've been seeing this for some time too. For me it's happening when the pod is evicted due to memory pressure:
43m Warning Evicted pod/redis-node-2 The node was low on resource: memory. Threshold quantity: 750Mi, available: 764316Ki.
The logs from the pod:
1:M 21 Feb 2024 23:23:23.596 * Connection with replica redis-node-0.redis-headless.live-tdm.svc.cluster.local:6379 lost.
1:X 21 Feb 2024 23:23:23.922 * Sentinel new configuration saved on disk
1:X 21 Feb 2024 23:23:23.922 # +new-epoch 2
1:M 21 Feb 2024 23:23:24.586 * Connection with replica redis-node-1.redis-headless.live-tdm.svc.cluster.local:6379 lost.
1:X 21 Feb 2024 23:23:24.587 # +config-update-from sentinel 2a09ba7abbb41ee71e79087310d75f9809c3c815 redis-node-0.redis-headless.live-tdm.svc.cluster.local 26379 @ mymaster redis-node-2.redis-headless.live-tdm.svc.cluster.local 6379
1:X 21 Feb 2024 23:23:24.587 # +switch-master mymaster redis-node-2.redis-headless.live-tdm.svc.cluster.local 6379 redis-node-0.redis-headless.live-tdm.svc.cluster.local 6379
1:X 21 Feb 2024 23:23:24.589 * +slave slave redis-node-1.redis-headless.live-tdm.svc.cluster.local:6379 redis-node-1.redis-headless.live-tdm.svc.cluster.local 6379 @ mymaster redis-node-0.redis-headless.live-tdm.svc.cluster.local 6379
1:X 21 Feb 2024 23:23:24.589 * +slave slave redis-node-2.redis-headless.live-tdm.svc.cluster.local:6379 redis-node-2.redis-headless.live-tdm.svc.cluster.local 6379 @ mymaster redis-node-0.redis-headless.live-tdm.svc.cluster.local 6379
1:X 21 Feb 2024 23:23:24.602 * Sentinel new configuration saved on disk
1:signal-handler (1708557805) Received SIGTERM scheduling shutdown...
1:signal-handler (1708557805) Received SIGTERM scheduling shutdown...
1:X 21 Feb 2024 23:23:25.483 * User requested shutdown...
1:X 21 Feb 2024 23:23:25.484 # Sentinel is now ready to exit, bye bye...
1:M 21 Feb 2024 23:23:25.516 * User requested shutdown...
1:M 21 Feb 2024 23:23:25.516 # Redis is now ready to exit, bye bye...
The terminated pod remains in Completed state until i delete it. I would expect the pod to be removed and a new one started in it's place.
I'm running kubernetes 1.27.3 in AKS.
from charts.
Did you check the space of the cluster? It seems the pod had a problem with the memory... the PVCs show something?
from charts.
I'm glad you solved the issue.
from charts.
This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.
from charts.
Related Issues (20)
- [bitnami/elasticsearch] Unable to open ingress access(TLS) HOT 6
- [bitnami/keycloak] is external infinispan cache configuration supported? HOT 4
- [bitnami/mongodb] backoffLimit should be configurable in backup cronjob HOT 1
- [bitnami/redis] error calling concat: runtime error: invalid memory address or nil pointer dereference HOT 1
- [bitnami/thanos] thanos distributor pods goes in crashloopback state. HOT 4
- The script with empty input HOT 3
- [bitnami/oauth2-proxy] subchart redis.replica.replicaCount isn't working HOT 3
- Consumers can't connect - Bootstrap broker disconnected HOT 1
- [bitnami/thanos] thanos helm chart renders strange hostname for sidecarsService dnsDiscovery HOT 7
- [bitnami/postgresql] existingExtendedConfigmap and extendedConfiguration broken since chart v14.2.1 HOT 6
- [bitnami/external-dns] pihole.tlsSkipVerify: true gives an error HOT 3
- [bitnami/elasticsearch] in Dualstack IPv6 & SVC mismatch
- Schema Registry with external Confluent Kafka HOT 1
- Wrong selector in Networkpolicy for Argo Workflows HOT 3
- thanos helm chart does not render with existingObjstoreSecret HOT 11
- Inconsistent defaults for PodMonitor / ServiceMonitor - interval / namespace should be left empty HOT 4
- I want to connect to Redis Sentinel from outside k8s. HOT 6
- [bitnami/schema-registry] Change default file mode to 0600 so that container crash can be recovered HOT 1
- Redis Helm charts not restarted on secret change? HOT 2
- [bitnami/mlflow] GCS Support for MLFlow HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from charts.