How to remove a failed TiKV store when using local storage about tidb-operator HOT 4 CLOSED

After playing around it looks like the correct way to handle this scenario is to:

1. Enable the advanced statefulset when installing the operator
2. To remove basic-tikv-2, edit the TidbCluster CR and change tikv.statefulset.replicas to the desired number (5) and set the annotations.tikv.tidb.pingcap.com/delete-slots: '[2]'.

Note the documentation here says to set tikv.replicas to the 5, which doesn't help in this scenario.

Would someone be able to confirm what I've just written? I think it would be useful to update the documentation to detail how to remove a failed pods.

from tidb-operator.

To remove a failed TiKV, if you don't want to use Advanced StatefulSet, the correct way is:

I can do this by removing the PVC for tikv-2, removing the store using pd-ctl unsafe remove-failed-stores and then restarting basic-tikv-2.

If you want to use Advanced StatefulSet, the correct way is:

Enable the advanced statefulset when installing the operator; set the annotations.tikv.tidb.pingcap.com/delete-slots: '[2]'

There is no need to change the tikv.replicas.

Note the documentation here says to set tikv.replicas to the 5, which doesn't help in this scenario.

Do you have any details about this?

from tidb-operator.

To remove a failed TiKV, if you don't want to use Advanced StatefulSet, the correct way is...:
Apologies! I've just retried this and setting the annotation does seem to do the correct thing. Maybe I didn't wait long enough.

One further question. If I am using TiKV with local storage and I remove a failed TiKV via the Advanced StatefulSet annotation should I also use pd-ctl unsafe remove-failed-stores to remove the store? If so is it better to use remove-failed-store before or after setting the statefulset annotation?

Thanks.

from tidb-operator.

If I am using TiKV with local storage and I remove a failed TiKV via the Advanced StatefulSet annotation should I also use pd-ctl unsafe remove-failed-stores to remove the store? If so is it better to use remove-failed-store before or after setting the statefulset annotation?

You can use remove-failed-store after setting the statefulset annotation

from tidb-operator.