Followers falls out of sync but reports they are perfectly in sync if leader dies and comes back empty about tile38 HOT 2 OPEN

I'm able to replicate it on AWS ECS (any container environment should do)

1. Setup configuration with 1 leader and 2 followers.
2. When setting up the follow, use a DNS record to locate the leader (this is the important part).
3. Startup the servers and begin writing keys and generally futzing with it. Make sure you write at least one OBJECT and one STRING and issue SET and DEL and DROP so that it appears in the AOF. Before the next step make sure a STRING and a OBJECT type exist.
4. kill the leader with kill -9 <pid>
5. ECS/Docker should see the task died and reboot the container. If it doesn't, do so manually. Since there is no data attached it should reboot with a clean /data directory and reinitialize based on your config.
6. The leader should come online and reattach to followers but with an empty AOF/DB.
7. The followers report that everything is all good even though they are no longer in sync with each other as the master is empty and the followers have records.
8. Begin writing records and notice that the followers attempt to stay in sync but don't really as the old keys are never cleared.
9. Repeat the kill on one of the followers and notice that it now comes in sync with the leader properly as it downloads a new AOF.
10. At this point it is in the state i described above, the leader and one of the followers are in sync but the last remaining follower is holding onto old records.

So what really happened here is that the leader will not cleanly killed and when it comes back online, its empty. The followers don't notice this change and continue along thinking everything is good. This means the issue is that the IP address changed of the leader during the reboot and the followers didn't re-verify that they weren't connecting to a leader who's AOF didn't match theirs (or even their server id).

In my case the leader was dying because AOFSHRINK was not properly running so it ran out of drive space. Its a solvable problem but still reveals that there is an issue.

from tile38.

I can confirm this on my side. Normally, immediately after connecting to a leader, a follower will issue some md5 checks to the leader to determine if they share the same AOF, and it not the follower AOF should be reset to match the leader.
I'll need to dig a little deeper, but I suspect the hard reset of the leader, which changes the server_id and empties the AOF, may be confusing the followers.

from tile38.