[NEW] Trigger manual failover on SIGTERM to primary (cluster) about valkey HOT 3 OPEN

This is a good idea. In my fork, i have a similar feature like this (detect a disk error and do failover). The basic idea is that, the primary will pick a best replica and send a CLUSTER FAILOVER to it (and wait for it). Do you like this approach or need me to try it?

from valkey.

the primary will pick a best replica and send a CLUSTER FAILOVER to it (and wait for it). Do you like this approach

Yes, this is strait-forward. I like it.

I have another idea, similar but maybe faster(?) but more complex(?). This is the idea: The primary first pauses writes, then waits for the replica to replicate everything and then sends CLUSTER FAILOVER FORCE. This avoids step 1 below. This is from the docs of CLUSTER FAILOVER:

1. The replica tells the master to stop processing queries from clients.

2. The master replies to the replica with the current replication offset.

3. The replica waits for the replication offset to match on its side, to make sure it processed all the data from the master before it continues.

4. The replica starts a failover, obtains a new configuration epoch from the majority of the masters, and broadcasts the new configuration.

5. The old master receives the configuration update: unblocks its clients and starts replying with redirection messages so that they’ll continue the chat with the new master.

And for FORCE:

If the FORCE option is given, the replica does not perform any handshake with the master, that may be not reachable, but instead just starts a failover ASAP starting from point 4. This is useful when we want to start a manual failover while the master is no longer reachable.

from valkey.

The primary first pauses writes, then waits for the replica to replicate everything and then sends CLUSTER FAILOVER FORCE.

yeah, this seems ok to me, faster.

This different i think is, one is that the replica thinks the offset is ok and start the failover, and the other is that the primary tells the replica that it can start the failover.

The CLUSTER FAILOVER one:

1. Primary detect a SIGTERM and then pick a best replica to send CLUSTER FAILOVER in serverCron, 100ms a time.
2. Replica receives a CLUSTER FAILOVER and tells primary to stop the write (and repsonse the offset), and wait the offset become ok. (in clusterCron, 100ms a time)
3. Replica start the failover

primary serverCron, primary send CLUSTER FAILOVER, replcia send a MFSTART, primary send a PING, replica clusterCron and start the failover.
100ms + a command + a mfstart + a ping + 100ms

The CLUSTER FAILOVER FORCE one:

1. Primary detect a SIGTERM and then stop the write, and then send the REPLCONF GETACK to all replicas and wait the response.
2. Primary receives the REPLCONF ACK, check replica->repl_ack_off and primary_repl_offset, if match, send the CLUSTER FAILOVER FORCE to the replica.
3. Replica start the failover.

primary serverCron, primary send REPLCONF GETACK, replica send REPLCONF ACK, primary send CLUSTER FAILOVER FORCE, replica clusterCron and start the failover
100ms + a command + a command + a command + 100ms

from valkey.