PD automatic failover about tidb-operator HOT 2 CLOSED

I am working hard on the feature. I encountered an issue during test verification:

PD's StatefulSet podManagementPolicy is OrderedReady, so if the 0 ordinal Pod was not ready, the StatefulSet can't be scaled out, which means can't failover.

I will change the pd podManagementPolicy to Parallel before the pd failover. Which will be another pr.

from tidb-operator.

Another issue: when failover with current pd start script, the ordinal 0 pd will start as a new pd cluster, because it will start with initial-cluster option and new empty data dir when the ordinal is 0.

Solution:

Change the pd podManagementPolicy to Parallel

Add an annotation key bootstrapping to PD's ordinal 0 Pod to indicate that the cluster is bootstrapping, and a replicas annotation to indicate the replicas of current pd cluster.

• if there are no PVCs that ordinal is greater than 0, set bootstrapping to true
• else set bootstrapping to false

Modify the pd start script

• if the ordinal is greater than 0, verify the cluster is working now before starting when cluster bootstrapping
• if the ordinal is 0 and bootstrapping is true, use initial-cluster option to start this member, else use join option to start this member instead.

@tennix @onlymellb @xiaojingchen PTAL

from tidb-operator.