Code Monkey home page Code Monkey logo

Comments (6)

mmosierteamvelocity avatar mmosierteamvelocity commented on September 23, 2024 1

It looks like it's related to a bug in Kubernetes version 1.27.

Setting a mirror pod's phase to Succeeded or Failed can prevent the corresponding static pod from restarting due mutation of a Kubelet cache. (https://github.com/kubernetes/kubernetes/pull/116482, [@smarterclayton](https://github.com/smarterclayton)) [SIG Node]

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.28.md

After upgrading to 1.28 we have not seen this issue resurface.

from charts.

Mauraza avatar Mauraza commented on September 23, 2024

Hi @mmosierteamvelocity,

Did you check if some event shows in the pod that the status was "succeeded"?

from charts.

paulclark222 avatar paulclark222 commented on September 23, 2024

@mmosierteamvelocity
@Mauraza

I've been seeing this for some time too. For me it's happening when the pod is evicted due to memory pressure:

43m         Warning   Evicted                  pod/redis-node-2                                           The node was low on resource: memory. Threshold quantity: 750Mi, available: 764316Ki.

The logs from the pod:

1:M 21 Feb 2024 23:23:23.596 * Connection with replica redis-node-0.redis-headless.live-tdm.svc.cluster.local:6379 lost.
1:X 21 Feb 2024 23:23:23.922 * Sentinel new configuration saved on disk
1:X 21 Feb 2024 23:23:23.922 # +new-epoch 2
1:M 21 Feb 2024 23:23:24.586 * Connection with replica redis-node-1.redis-headless.live-tdm.svc.cluster.local:6379 lost.
1:X 21 Feb 2024 23:23:24.587 # +config-update-from sentinel 2a09ba7abbb41ee71e79087310d75f9809c3c815 redis-node-0.redis-headless.live-tdm.svc.cluster.local 26379 @ mymaster redis-node-2.redis-headless.live-tdm.svc.cluster.local 6379
1:X 21 Feb 2024 23:23:24.587 # +switch-master mymaster redis-node-2.redis-headless.live-tdm.svc.cluster.local 6379 redis-node-0.redis-headless.live-tdm.svc.cluster.local 6379
1:X 21 Feb 2024 23:23:24.589 * +slave slave redis-node-1.redis-headless.live-tdm.svc.cluster.local:6379 redis-node-1.redis-headless.live-tdm.svc.cluster.local 6379 @ mymaster redis-node-0.redis-headless.live-tdm.svc.cluster.local 6379
1:X 21 Feb 2024 23:23:24.589 * +slave slave redis-node-2.redis-headless.live-tdm.svc.cluster.local:6379 redis-node-2.redis-headless.live-tdm.svc.cluster.local 6379 @ mymaster redis-node-0.redis-headless.live-tdm.svc.cluster.local 6379
1:X 21 Feb 2024 23:23:24.602 * Sentinel new configuration saved on disk
1:signal-handler (1708557805) Received SIGTERM scheduling shutdown...
1:signal-handler (1708557805) Received SIGTERM scheduling shutdown...
1:X 21 Feb 2024 23:23:25.483 * User requested shutdown...
1:X 21 Feb 2024 23:23:25.484 # Sentinel is now ready to exit, bye bye...
1:M 21 Feb 2024 23:23:25.516 * User requested shutdown...
1:M 21 Feb 2024 23:23:25.516 # Redis is now ready to exit, bye bye...

The terminated pod remains in Completed state until i delete it. I would expect the pod to be removed and a new one started in it's place.

I'm running kubernetes 1.27.3 in AKS.

from charts.

Mauraza avatar Mauraza commented on September 23, 2024

Hi @mmosierteamvelocity,

Did you check the space of the cluster? It seems the pod had a problem with the memory... the PVCs show something?

from charts.

Mauraza avatar Mauraza commented on September 23, 2024

Hi @mmosierteamvelocity,

I'm glad you solved the issue.

from charts.

github-actions avatar github-actions commented on September 23, 2024

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

from charts.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.