Modify tikv config but not rolling update pods about tidb-operator HOT 10 CLOSED

pvcModifier.Sync is called after tikvMemberManager.Sync, is there any other log about why TiKV still in the Upgrade phase but without re-creating Pods?

from tidb-operator.

And here are two things to be figure out

1. Why I modify tikv config, but pods are not rolling?
2. How can I fix these error message?

from tidb-operator.

I found the reason are related to these codes
https://github.com/pingcap/tidb-operator/blob/master/pkg/manager/member/tikv_member_manager.go#L823-L831

	// Scaling takes precedence over upgrading.
	if tc.TiKVStsDesiredReplicas() != *set.Spec.Replicas {
		tc.Status.TiKV.Phase = v1alpha1.ScalePhase
	} else if upgrading && tc.Status.PD.Phase != v1alpha1.UpgradePhase {
		if !tc.IsComponentLeaderEvicting(v1alpha1.TiKVMemberType) { // skip upgrade if someone is evicting leader
			tc.Status.TiKV.Phase = v1alpha1.UpgradePhase
		}
	} else {
		tc.Status.TiKV.Phase = v1alpha1.NormalPhase
	}

upgrading is always true, so the tc.Status.TiKV.Phase can't be assgin v1alpha1.NormalPhase

The reason of upgrading == true is some pods' revision are not equal to cluster revision like tikv-1

Name:             kvstore-prod-video-at-tikv-1
Namespace:        di-kvstore
Priority:         0
Service Account:  default
Node:             x 
Start Time:       Fri, 15 Dec 2023 17:38:34 +0800
Labels:           app.kubernetes.io/component=tikv
                  app.kubernetes.io/instance=kvstore-prod-video-at
                  app.kubernetes.io/managed-by=tidb-operator
                  app.kubernetes.io/name=tidb-cluster
                  cluster=kvstore-prod-video-at
                  controller-revision-hash=kvstore-prod-video-at-tikv-6978bbb55d
                  env=prod
                  statefulset.kubernetes.io/pod-name=kvstore-prod-video-at-tikv-1
                  tidb.pingcap.com/cluster-id=7312730664965344747
                  tidb.pingcap.com/store-id=18

but now cluster revision is

    Stateful Set:
      Collision Count:      0
      Current Replicas:     12
      Current Revision:     kvstore-prod-video-at-tikv-77565ff546
      Observed Generation:  13
      Ready Replicas:       30
      Replicas:             30
      Update Revision:      kvstore-prod-video-at-tikv-77565ff546
      Updated Replicas:     12

from tidb-operator.

Modify tikv config, then scale out cluster, then restart some tikv pods.

How did you restart TiKV pods? have you applied another operation before the previous has been done?

What's the value of .Spec.UpdateStrategy.RollingUpdate.Partition for the TiKV StatefulSet now?

from tidb-operator.

Modify tikv config, then scale out cluster, then restart some tikv pods.

How did you restart TiKV pods? have you applied another operation before the previous has been done?

What's the value of .Spec.UpdateStrategy.RollingUpdate.Partition for the TiKV StatefulSet now?

Oh, I found this error happened before I did these operations.

 Update Strategy:
    Rolling Update:
      Partition:  30

from tidb-operator.

could you try to update the partition field to the max ordinal of not updated pods to manually trigger update?

from tidb-operator.

I found restart pod can update revision, can I manually restart all pods by using

    annotations:
      tidb.pingcap.com/restartedAt: 2020-04-20T12:00

from tidb-operator.

But I still want to figure out why this error happened, is there any way to fix it?

from tidb-operator.

But I still want to figure out why this error happened, is there any way to fix it?

I think it may be caused by extra modification before the previous has been done, then the Revision and Partition of the StatefulSet is broken.

from tidb-operator.

fixed this problem after restarting cluster

from tidb-operator.