Comments (5)
Can you show the size of your PVCs (both TiKV and PD)?
from tidb-operator.
The problem happens on create too, not just on update. I created a new TidbCluster and here's all the manifests and logs:
I created TidbCluster CR with the same storage sizes as before (15675Mi and 1740Mi). Here's the CR copied from Lens after creation:
apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
creationTimestamp: '2024-05-20T06:26:04Z'
generation: 16
labels:
lexoya/component: '25'
lexoya/name: wtiagtmv
lexoya/version: '69'
name: wtiagtmv
namespace: rvgnrvpp
resourceVersion: '572028224'
uid: 77ce4dd6-fb5d-4814-9ca7-96393bec6d94
selfLink: /apis/pingcap.com/v1alpha1/namespaces/rvgnrvpp/tidbclusters/wtiagtmv
status:
clusterID: '7370964382394004606'
conditions:
- lastTransitionTime: '2024-05-20T06:28:23Z'
lastUpdateTime: '2024-05-20T06:28:23Z'
message: TiDB cluster is fully up and running
reason: Ready
status: 'True'
type: Ready
pd:
image: repo.lexoya.com/pingcap/pd:v7.5.1
leader:
clientURL: http://wtiagtmv-pd-0.wtiagtmv-pd-peer.rvgnrvpp.svc:2379
health: true
id: '10249660128629638972'
lastTransitionTime: '2024-05-20T06:26:27Z'
name: wtiagtmv-pd-0
members:
wtiagtmv-pd-0:
clientURL: http://wtiagtmv-pd-0.wtiagtmv-pd-peer.rvgnrvpp.svc:2379
health: true
id: '10249660128629638972'
lastTransitionTime: '2024-05-20T06:26:27Z'
name: wtiagtmv-pd-0
phase: Normal
statefulSet:
collisionCount: 0
currentReplicas: 1
currentRevision: wtiagtmv-pd-6597dfcbd4
observedGeneration: 1
readyReplicas: 1
replicas: 1
updateRevision: wtiagtmv-pd-6597dfcbd4
updatedReplicas: 1
synced: true
volumes:
pd:
boundCount: 1
currentCapacity: 2Gi
currentCount: 1
currentStorageClass: csi-cephfs-sc-nvme
modifiedCapacity: 1740Mi
modifiedCount: 0
modifiedStorageClass: csi-cephfs-sc-nvme
name: pd
resizedCapacity: 1740Mi
resizedCount: 0
pump: {}
ticdc: {}
tidb:
image: repo.lexoya.com/pingcap/tidb:v7.5.1
members:
wtiagtmv-tidb-0:
health: true
lastTransitionTime: '2024-05-20T06:28:23Z'
name: wtiagtmv-tidb-0
node: 192.168.110.158
phase: Normal
statefulSet:
collisionCount: 0
currentReplicas: 1
currentRevision: wtiagtmv-tidb-55f66f78b4
observedGeneration: 1
readyReplicas: 1
replicas: 1
updateRevision: wtiagtmv-tidb-55f66f78b4
updatedReplicas: 1
tiflash: {}
tikv:
bootStrapped: true
conditions:
- lastTransitionTime: '2024-05-20T06:27:13Z'
message: Leader can be scheduled to all nodes
reason: NoLeaderEviction
status: 'False'
type: LeaderEvicting
image: repo.lexoya.com/pingcap/tikv:v7.5.1
phase: Normal
statefulSet:
collisionCount: 0
currentReplicas: 1
currentRevision: wtiagtmv-tikv-5464f9546f
observedGeneration: 1
readyReplicas: 1
replicas: 1
updateRevision: wtiagtmv-tikv-5464f9546f
updatedReplicas: 1
stores:
'1':
id: '1'
ip: wtiagtmv-tikv-0.wtiagtmv-tikv-peer.rvgnrvpp.svc
lastTransitionTime: '2024-05-20T06:27:12Z'
leaderCount: 60
podName: wtiagtmv-tikv-0
state: Up
synced: true
volumes:
tikv:
boundCount: 1
currentCapacity: 16Gi
currentCount: 1
currentStorageClass: csi-cephfs-sc-nvme
modifiedCapacity: 15675Mi
modifiedCount: 0
modifiedStorageClass: csi-cephfs-sc-nvme
name: tikv
resizedCapacity: 15675Mi
resizedCount: 0
tiproxy: {}
spec:
configUpdateStrategy: RollingUpdate
discovery:
labels:
lexoya/component: '25'
lexoya/name: wtiagtmv
lexoya/version: '69'
limits:
cpu: 50m
memory: 100Mi
requests:
cpu: 1m
memory: 20Mi
enableDynamicConfiguration: true
enablePVReclaim: false
helper:
image: repo.lexoya.com/library/alpine:3.16.0
imagePullPolicy: IfNotPresent
labels:
lexoya/component: '25'
lexoya/name: wtiagtmv
lexoya/version: '69'
pd:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/component
operator: In
values:
- pd
- key: lexoya/name
operator: In
values:
- wtiagtmv
topologyKey: kubernetes.io/hostname
weight: 1
baseImage: repo.lexoya.com/pingcap/pd
config: |
[dashboard]
internal-proxy = true
[log]
[log.file]
max-backups = 1
max-days = 1
max-size = 100
[replication]
max-replicas = 1
limits:
cpu: '1.318'
memory: 223M
maxFailoverCount: 3
mountClusterClientSecret: true
replicas: 1
requests:
cpu: 180m
memory: 120Mi
storage: 1740Mi
startTimeout: 30
pvReclaimPolicy: Delete
startScriptVersion: v2
suspendAction: {}
tidb:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/component
operator: In
values:
- tidb
- key: lexoya/name
operator: In
values:
- wtiagtmv
topologyKey: kubernetes.io/hostname
weight: 1
baseImage: repo.lexoya.com/pingcap/tidb
config: |
[log]
[log.file]
max-backups = 1
max-days = 1
max-size = 100
[performance]
tcp-keep-alive = true
limits:
cpu: 467m
memory: 1276M
maxFailoverCount: 3
replicas: 1
requests:
cpu: 100m
memory: 200Mi
service:
type: ClusterIP
tikv:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/component
operator: In
values:
- tikv
- key: lexoya/name
operator: In
values:
- wtiagtmv
topologyKey: kubernetes.io/hostname
weight: 1
baseImage: repo.lexoya.com/pingcap/tikv
config: |
log-level = "info"
[log]
[log.file]
max-backups = 1
max-days = 1
max-size = 100
[raftdb]
max-total-wal-size = "1GB"
[rocksdb]
max-total-wal-size = "1GB"
[storage]
reserve-space = "100MB"
limits:
cpu: 467m
memory: 4511M
maxFailoverCount: 3
mountClusterClientSecret: true
replicas: 1
requests:
cpu: 100m
memory: 1536Mi
storage: 15675Mi
scalePolicy:
scaleInParallelism: 1
scaleOutParallelism: 1
timezone: UTC
tlsCluster: {}
version: v7.5.1
TiKV pvc:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: tikv-wtiagtmv-tikv-0
namespace: rvgnrvpp
uid: 9d15043b-0636-45bb-9a1b-b1af1ab02948
resourceVersion: '572027744'
creationTimestamp: '2024-05-20T06:26:28Z'
labels:
app.kubernetes.io/component: tikv
app.kubernetes.io/instance: wtiagtmv
app.kubernetes.io/managed-by: tidb-operator
app.kubernetes.io/name: tidb-cluster
tidb.pingcap.com/cluster-id: '7370964382394004606'
tidb.pingcap.com/pod-name: wtiagtmv-tikv-0
tidb.pingcap.com/store-id: '1'
annotations:
pv.kubernetes.io/bind-completed: 'yes'
pv.kubernetes.io/bound-by-controller: 'yes'
tidb.pingcap.com/pod-name: wtiagtmv-tikv-0
volume.beta.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
volume.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
finalizers:
- kubernetes.io/pvc-protection
selfLink: /api/v1/namespaces/rvgnrvpp/persistentvolumeclaims/tikv-wtiagtmv-tikv-0
status:
phase: Bound
accessModes:
- ReadWriteOnce
capacity:
storage: 16Gi
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 15675Mi
volumeName: pvc-9d15043b-0636-45bb-9a1b-b1af1ab02948
storageClassName: csi-cephfs-sc-nvme
volumeMode: Filesystem
PD pvc:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pd-wtiagtmv-pd-0
namespace: rvgnrvpp
uid: ea461780-f79f-47b8-947b-f2701ce5d192
resourceVersion: '572027731'
creationTimestamp: '2024-05-20T06:26:06Z'
labels:
app.kubernetes.io/component: pd
app.kubernetes.io/instance: wtiagtmv
app.kubernetes.io/managed-by: tidb-operator
app.kubernetes.io/name: tidb-cluster
tidb.pingcap.com/cluster-id: '7370964382394004606'
tidb.pingcap.com/member-id: '10249660128629638972'
tidb.pingcap.com/pod-name: wtiagtmv-pd-0
annotations:
pv.kubernetes.io/bind-completed: 'yes'
pv.kubernetes.io/bound-by-controller: 'yes'
tidb.pingcap.com/pod-name: wtiagtmv-pd-0
volume.beta.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
volume.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
finalizers:
- kubernetes.io/pvc-protection
selfLink: /api/v1/namespaces/rvgnrvpp/persistentvolumeclaims/pd-wtiagtmv-pd-0
status:
phase: Bound
accessModes:
- ReadWriteOnce
capacity:
storage: 2Gi
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1740Mi
volumeName: pvc-ea461780-f79f-47b8-947b-f2701ce5d192
storageClassName: csi-cephfs-sc-nvme
volumeMode: Filesystem
operator logs:
E0520 06:26:05.033840 1 tidb_initializer_controller.go:106] TiDBInitializer: rvgnrvpp/wtiagtmv, sync failed, err: TidbInitManager.Sync: failed to get tidbcluster wtiagtmv for TidbInitializer rvgnrvpp/wtiagtmv, error: tidbcluster.pingcap.com "wtiagtmv" not found, requeuing
I0520 06:26:05.318603 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbNGMonitoring", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"815b5ddc-a8c3-44a1-bbf7-2cd0420030c3", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026966", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Secret/wtiagtmv-tc-client-tls for controller TidbNGMonitoring/wtiagtmv successfully
I0520 06:26:05.318695 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Role/wtiagtmv-discovery for controller TidbCluster/wtiagtmv successfully
I0520 06:26:05.319347 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbDashboard", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"b39fbc16-cdb9-4a15-bf3a-c7a2e02f6d03", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026967", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-tidb-dashboard-exposed in wtiagtmv successful
I0520 06:26:05.353237 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ServiceAccount/wtiagtmv-discovery for controller TidbCluster/wtiagtmv successfully
I0520 06:26:05.357424 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbDashboard", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"b39fbc16-cdb9-4a15-bf3a-c7a2e02f6d03", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026967", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet wtiagtmv-tidb-dashboard in wtiagtmv successful
I0520 06:26:05.365138 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbNGMonitoring", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"815b5ddc-a8c3-44a1-bbf7-2cd0420030c3", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026966", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-ng-monitoring in wtiagtmv successful
I0520 06:26:05.393694 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbNGMonitoring", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"815b5ddc-a8c3-44a1-bbf7-2cd0420030c3", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026966", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-ng-monitoring-3961393 for controller TidbNGMonitoring/wtiagtmv successfully
I0520 06:26:05.417535 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbNGMonitoring", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"815b5ddc-a8c3-44a1-bbf7-2cd0420030c3", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026966", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet wtiagtmv-ng-monitoring in wtiagtmv successful
I0520 06:26:05.512125 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create RoleBinding/wtiagtmv-discovery for controller TidbCluster/wtiagtmv successfully
I0520 06:26:05.544550 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Deployment/wtiagtmv-discovery for controller TidbCluster/wtiagtmv successfully
I0520 06:26:05.596824 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Service/wtiagtmv-discovery for controller TidbCluster/wtiagtmv successfully
I0520 06:26:05.859607 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-pd in wtiagtmv successful
I0520 06:26:06.061752 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbInitializer", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"1fc1a34b-60d5-4c97-823c-6d2358e6378b", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026963", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-tidb-initializer for controller TidbInitializer/wtiagtmv successfully
E0520 06:26:06.088485 1 tidb_initializer_controller.go:106] TiDBInitializer: rvgnrvpp/wtiagtmv, sync failed, err: updateStatus: failed to get job wtiagtmv-tidb-initializer for TidbInitializer rvgnrvpp/wtiagtmv, error: job.batch "wtiagtmv-tidb-initializer" not found, requeuing
I0520 06:26:06.088665 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbInitializer", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"1fc1a34b-60d5-4c97-823c-6d2358e6378b", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026963", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Job/wtiagtmv-tidb-initializer for controller TidbInitializer/wtiagtmv successfully
I0520 06:26:06.242077 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-pd-peer in wtiagtmv successful
I0520 06:26:06.273267 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-pd-6334666 for controller TidbCluster/wtiagtmv successfully
I0520 06:26:06.301665 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet wtiagtmv-pd in wtiagtmv successful
I0520 06:26:06.349515 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
I0520 06:26:06.349721 1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for PD cluster running, requeuing
E0520 06:26:06.813751 1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp: lookup wtiagtmv-pd.rvgnrvpp on 10.43.0.10:53: no such host, service rvgnrvpp/wtiagtmv-pd has no endpoints
I0520 06:26:06.943683 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
E0520 06:26:06.943885 1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
E0520 06:26:07.348161 1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp: lookup wtiagtmv-pd.rvgnrvpp on 10.43.0.10:53: no such host, service rvgnrvpp/wtiagtmv-pd has no endpoints
E0520 06:26:07.387432 1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
E0520 06:26:07.723528 1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp 10.43.137.11:2379: connect: connection refused, service rvgnrvpp/wtiagtmv-pd has no endpoints
E0520 06:26:07.761553 1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
I0520 06:26:08.125232 1 tidb_init_manager.go:142] TidbInitializer: [rvgnrvpp/wtiagtmv] updated successfully
{"level":"warn","ts":"2024-05-20T06:26:10.220Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-23807697-7bf5-4485-93c1-93cb67f81e10/wtiagtmv-pd.rvgnrvpp:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 10.43.137.11:2379: connect: connection refused\""}
E0520 06:26:10.220875 1 monitor_manager.go:786] context deadline exceeded
E0520 06:26:10.220905 1 monitor_manager.go:107] Fail to sync TiDB Dashboard metrics config for TiDB cluster [rvgnrvpp/wtiagtmv], error: context deadline exceeded
I0520 06:26:10.230130 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Secret/tidbmonitor-wtiagtmv-tls-assets for controller TidbMonitor/wtiagtmv successfully
I0520 06:26:10.265579 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-prometheus in wtiagtmv successful
I0520 06:26:10.300232 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-monitor-reloader in wtiagtmv successful
I0520 06:26:10.373574 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-grafana in wtiagtmv successful
I0520 06:26:10.400498 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-monitor for controller TidbMonitor/wtiagtmv successfully
I0520 06:26:10.414949 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-monitor-grafana for controller TidbMonitor/wtiagtmv successfully
I0520 06:26:10.431792 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Secret/wtiagtmv-monitor for controller TidbMonitor/wtiagtmv successfully
I0520 06:26:10.450916 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ServiceAccount/wtiagtmv-monitor for controller TidbMonitor/wtiagtmv successfully
I0520 06:26:10.463887 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Role/wtiagtmv-monitor for controller TidbMonitor/wtiagtmv successfully
I0520 06:26:10.512205 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create RoleBinding/wtiagtmv-monitor for controller TidbMonitor/wtiagtmv successfully
W0520 06:26:10.544937 1 warnings.go:70] spec.template.spec.initContainers[0].env[5].name: duplicate name "TZ"
I0520 06:26:10.545723 1 tidb_monitor_controller.go:99] TidbMonitor: rvgnrvpp/wtiagtmv, still need sync: TidbMonitor: [rvgnrvpp/wtiagtmv], waiting for tidbmonitor running, requeuing
I0520 06:26:10.545752 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet wtiagtmv-monitor in wtiagtmv successful
I0520 06:26:10.545822 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Warning' reason: 'FailedSync' Sync TidbMonitor[rvgnrvpp/wtiagtmv] Statefulset failed, err:TidbMonitor: [rvgnrvpp/wtiagtmv], waiting for tidbmonitor running
E0520 06:26:11.662535 1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp 10.43.137.11:2379: connect: connection refused, service rvgnrvpp/wtiagtmv-pd has no endpoints
E0520 06:26:11.692055 1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
E0520 06:26:12.243252 1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp 10.43.137.11:2379: connect: connection refused, service rvgnrvpp/wtiagtmv-pd has no endpoints
E0520 06:26:12.282986 1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
E0520 06:26:15.031799 1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp 10.43.137.11:2379: connect: connection refused
I0520 06:26:15.374715 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
E0520 06:26:15.374880 1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
I0520 06:26:15.506073 1 tidb_dashboard_control.go:134] TidbDashboard: [rvgnrvpp/wtiagtmv], update status successfully
E0520 06:26:15.689451 1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp 10.43.137.11:2379: connect: connection refused
E0520 06:26:15.729608 1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
{"level":"warn","ts":"2024-05-20T06:26:16.546Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-23807697-7bf5-4485-93c1-93cb67f81e10/wtiagtmv-pd.rvgnrvpp:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 10.43.137.11:2379: connect: connection refused\""}
E0520 06:26:16.547071 1 monitor_manager.go:786] context deadline exceeded
E0520 06:26:16.547108 1 monitor_manager.go:107] Fail to sync TiDB Dashboard metrics config for TiDB cluster [rvgnrvpp/wtiagtmv], error: context deadline exceeded
I0520 06:26:16.859702 1 pv_control.go:192] PV: [pvc-e9d36715-8a9e-41d5-aec7-5cd142c33e46] updated successfully, : rvgnrvpp/wtiagtmv
I0520 06:26:16.893560 1 tidb_monitor_control.go:120] TidbMonitor: [rvgnrvpp/wtiagtmv] updated successfully
I0520 06:26:20.736268 1 tidb_ng_monitoring_control.go:153] TidbNGMonitoring: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:26:27.956642 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
I0520 06:26:28.002556 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027294", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-tikv-peer in wtiagtmv successful
I0520 06:26:28.032478 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027294", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-tikv-6664313 for controller TidbCluster/wtiagtmv successfully
I0520 06:26:28.060629 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027294", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet wtiagtmv-tikv in wtiagtmv successful
I0520 06:26:28.121069 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
I0520 06:26:28.121242 1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:26:28.624818 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
I0520 06:26:28.665325 1 tikv_member_manager.go:851] TiKV of Cluster rvgnrvpp/wtiagtmv not bootstrapped yet
I0520 06:26:28.696456 1 tikv_member_manager.go:965] TiKV of Cluster rvgnrvpp/wtiagtmv is not bootstrapped yet, no need to set store labels
I0520 06:26:28.763600 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
I0520 06:26:28.763730 1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:26:29.094536 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
I0520 06:26:29.135019 1 tikv_member_manager.go:851] TiKV of Cluster rvgnrvpp/wtiagtmv not bootstrapped yet
I0520 06:26:29.187116 1 tikv_member_manager.go:965] TiKV of Cluster rvgnrvpp/wtiagtmv is not bootstrapped yet, no need to set store labels
I0520 06:26:29.188874 1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:26:34.048099 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
I0520 06:26:34.112530 1 tikv_member_manager.go:851] TiKV of Cluster rvgnrvpp/wtiagtmv not bootstrapped yet
I0520 06:26:34.149937 1 tikv_member_manager.go:965] TiKV of Cluster rvgnrvpp/wtiagtmv is not bootstrapped yet, no need to set store labels
I0520 06:26:34.213674 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
I0520 06:26:34.213802 1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:26:34.648952 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
I0520 06:26:34.707919 1 tikv_member_manager.go:851] TiKV of Cluster rvgnrvpp/wtiagtmv not bootstrapped yet
I0520 06:26:34.745607 1 tikv_member_manager.go:965] TiKV of Cluster rvgnrvpp/wtiagtmv is not bootstrapped yet, no need to set store labels
I0520 06:26:34.747280 1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:26:42.170709 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:26:42.212448 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:26:42.310081 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
I0520 06:26:42.310189 1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:26:42.596586 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:26:42.641823 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:26:42.686692 1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:27:12.215575 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:12.263871 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:27:12.325857 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027531", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-tidb-peer in wtiagtmv successful
I0520 06:27:12.375332 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027531", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-tidb in wtiagtmv successful
I0520 06:27:12.401228 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027531", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-tidb-6436643 for controller TidbCluster/wtiagtmv successfully
I0520 06:27:12.433168 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027531", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet wtiagtmv-tidb in wtiagtmv successful
I0520 06:27:12.558263 1 pv_control.go:192] PV: [pvc-ea461780-f79f-47b8-947b-f2701ce5d192] updated successfully, : rvgnrvpp/wtiagtmv
I0520 06:27:12.926823 1 pv_control.go:192] PV: [pvc-9d15043b-0636-45bb-9a1b-b1af1ab02948] updated successfully, : rvgnrvpp/wtiagtmv
W0520 06:27:13.139695 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:13.140055 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:27:13.232600 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
E0520 06:27:13.232799 1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed [modify volumes for rvgnrvpp/wtiagtmv:tidb failed: component phase is not Normal, modify volumes for rvgnrvpp/wtiagtmv:tikv failed: try to stop evicting leader for tidbcluster rvgnrvpp/wtiagtmv], requeuing
W0520 06:27:13.529859 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:13.574867 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:13.829302 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:13.829678 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:27:13.983249 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
E0520 06:27:13.983419 1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed modify volumes for rvgnrvpp/wtiagtmv:tikv failed: try to stop evicting leader for tidbcluster rvgnrvpp/wtiagtmv, requeuing
W0520 06:27:14.283262 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:14.326888 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:14.427504 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:14.427768 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:27:14.555305 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:27:15.029004 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:15.076644 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:15.281218 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:15.281579 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:15.605736 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:15.677335 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:16.352707 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:16.352959 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:42.231406 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:42.281047 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:42.442824 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:42.443135 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:27:42.528256 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:27:42.818005 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:42.921079 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:43.010650 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:43.010925 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:27:43.084599 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:27:43.300293 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:43.335028 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:43.850035 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:43.850247 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:28:02.520831 1 tidb_monitor_control.go:120] TidbMonitor: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:28:08.451478 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:08.501857 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:08.611842 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:08.612097 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:28:08.707193 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:28:09.222522 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:09.261375 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:09.349822 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:09.350090 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:12.245023 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:12.291253 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:12.398556 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:12.398755 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:28:12.544318 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:28:12.893828 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:12.953947 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:13.058740 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:13.058939 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:23.202208 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:23.242786 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:23.352464 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:23.352565 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:28:23.421251 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:28:23.702868 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:23.743086 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:23.824304 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:23.824630 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:42.218587 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:42.266987 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:42.374751 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:42.374974 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:28:48.912223 1 tidb_init_manager.go:142] TidbInitializer: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:29:12.221467 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:29:12.273803 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:29:12.370491 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:29:12.370823 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:29:42.178022 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:29:42.220272 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:29:42.308498 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:29:42.308846 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
You can ignore logs for other CRs. The TidbInitializer only creates an "app" database and does nothing else.
As you can see, operator creates the cluster with correct pvc sizes but then tries to shrink from a higher size to the original correct size.
from tidb-operator.
from PVC
status:
phase: Bound
accessModes:
- ReadWriteOnce
capacity:
storage: 16Gi
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 15675Mi
It seems the CSI driver created a 16Gi volume instead of a 15675Mi one.
from tidb-operator.
from PVC
status: phase: Bound accessModes: - ReadWriteOnce capacity: storage: 16Gi spec: accessModes: - ReadWriteOnce resources: requests: storage: 15675MiIt seems the CSI driver created a 16Gi volume instead of a 15675Mi one.
Interesting. Maybe if operator looks at pvc spec.resources.requests.storage which is 15675Mi instead of pvc status.capacity.storage which is 16Gi, the problem is fixed. I'm not an expert but I think spec is a more solid thing to look at than status
from tidb-operator.
from PVC
status: phase: Bound accessModes: - ReadWriteOnce capacity: storage: 16Gi spec: accessModes: - ReadWriteOnce resources: requests: storage: 15675MiIt seems the CSI driver created a 16Gi volume instead of a 15675Mi one.
Interesting. Maybe if operator looks at pvc spec.resources.requests.storage which is 15675Mi instead of pvc status.capacity.storage which is 16Gi, the problem is fixed. I'm not an expert but I think spec is a more solid thing to look at than status
When reszing a PVC, operator need to check the status to ensure whether the expanding operation is finished.
from tidb-operator.
Related Issues (20)
- How to use Backup CR to backup tidb to aliyun OSS? HOT 4
- tikv upgrade blocked
- can't set MaxSkew, MinDomains or NodeAffinityPolicy for TopologySpreadConstraint
- Release v1.5.3
- tidb-scheduler pod keep crashing with TiDB Operator v1.5.3 HOT 3
- support setting extra command line arguments for TiDB
- BR backup could raise error when PD leader changed during BR initialization HOT 3
- BackupCR(7.5.1) can not read the correct size of backup in minio HOT 5
- backup: support multiple backup schedules
- Volume leaked if volume restore failed in restore volume step
- Release 1.6.0
- How to remove a failed TiKV store when using local storage HOT 4
- TiDB Cluster Does Not Survive K8s Upgrade (EKS) or Nodegroup Upgrade HOT 3
- TiDB Operator Crash HOT 1
- Documentation for operator features HOT 2
- Support PITR restore using --start-ts only
- If store number is 0, it is expected to set the volume backup to failed which will not blocked the whole clean schedule
- Can't change tiproxy service type HOT 7
- TiFlash没有暴露proxy的metrics HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tidb-operator.