Keepalived High Availability Issue: Both Nodes Selected as Master in RHOSP 16.2 Setup about keepalived HOT 3 CLOSED

please see more detail logs

SRINIVASAN THANDAPANI 16-04-2024 17:09 •
2024-04-16T11:38:15.462682+00:00 rtp-abc-02-6-0-sk Keepalived_vrrp[60022]: Assigned address fe80::f816:3eff:fe10:c8c6 for interface eth0
2024-04-16T11:38:15.462721+00:00 rtp-abc-02-6-0-sk Keepalived_vrrp[60022]: Registering gratuitous NDISC shared channel
2024-04-16T11:38:15.462777+00:00 rtp-abc-02-6-0-sk Keepalived_vrrp[60022]: (abc_control_eth_vrrp) removing VIPs.
2024-04-16T11:38:15.462828+00:00 rtp-abc-02-6-0-sk Keepalived_vrrp[60022]: VRRP sockpool: [ifindex( 2), family(IPv6), proto(112), fd(10,11)]
2024-04-16T11:38:15.524764+00:00 rtp-abc-02-6-0-sk Keepalived_vrrp[60022]: VRRP_Script(kad_check_drbd) succeeded
2024-04-16T11:38:15.605794+00:00 rtp-abc-02-6-0-sk Keepalived_vrrp[60022]: VRRP_Script(kad_check_abc) succeeded
2024-04-16T11:38:15.761381+00:00 rtp-abc-02-6-0-sk Keepalived_vrrp[60022]: VRRP_Script(kad_check_network) succeeded
2024-04-16T11:38:15.804316+00:00 rtp-abc-02-6-0-sk Keepalived_vrrp[60022]: VRRP_Script(kad_check_storage) succeeded
2024-04-16T11:38:15.804469+00:00 rtp-abc-02-6-0-sk Keepalived_vrrp[60022]: (abc_control_eth_vrrp) Entering BACKUP STATE
2024-04-16T11:38:15.888705+00:00 rtp-abc-02-6-0-sk abc_haagent.sh(13)[60059]: INFO: role: BACKUP
2024-04-16T11:38:33.832050+00:00 rtp-abc-02-6-0-sk Keepalived_vrrp[60022]: (abc_control_eth_vrrp) ### Receive advertisement timeout
2024-04-16T11:38:33.832390+00:00 rtp-abc-02-6-0-sk Keepalived_vrrp[60022]: (abc_control_eth_vrrp) Entering MASTER STATE
2024-04-16T11:38:33.832441+00:00 rtp-abc-02-6-0-sk Keepalived_vrrp[60022]: (abc_control_eth_vrrp) using locally configured advertisement interval (5000 milli-sec)
2024-04-16T11:38:33.832481+00:00 rtp-abc-02-6-0-sk Keepalived_vrrp[60022]: (abc_control_eth_vrrp) setting VIPs.
2024-04-16T11:38:33.832529+00:00 rtp-abc-02-6-0-sk Keepalived_vrrp[60022]: (abc_control_eth_vrrp) Sending/queueing Unsolicited Neighbour Adverts on eth0 for 2001:420:27c1:917::420
2024-04-16T11:38:33.832569+00:00 rtp-abc-02-6-0-sk Keepalived_vrrp[60022]: Sending unsolicited Neighbour Advert on eth0 for 2001:420:27c1:917::420
2024-04-16T11:38:33.832606+00:00 rtp-abc-02-6-0-sk Keepalived_vrrp[60022]: Sending unsolicited Neighbour Advert on eth0 for 2001:420:27c1:917::420
2024-04-16T11:38:33.832643+00:00 rtp-abc-02-6-0-sk Keepalived_vrrp[60022]: Sending unsolicited Neighbour Advert on eth0 for 2001:420:27c1:917::420
2024-04-16T11:38:33.832696+00:00 rtp-abc-02-6-0-sk Keepalived_vrrp[60022]: Sending unsolicited Neighbour Advert on eth0 for 2001:420:27c1:917::420
2024-04-16T11:38:33.832757+00:00 rtp-abc-02-6-0-sk Keepalived_vrrp[60022]: Sending unsolicited Neighbour Advert on eth0 for 2001:420:27c1:917::420
2024-04-16T11:38:33.917769+00:00 rtp-abc-02-6-0-sk abc_haagent.sh(13)[60231]: INFO: role: MASTER

from keepalived.

You give one configuration file, but do not make it clear whether this is the configuration on node 1 or on node 2. Can you please provide the configuration from the other node, and advise which node each configuration applies to.

Node 2 enters master state (after advising receive advertisment timeout) approximately 18 seconds after transitioning to backup state. That is 3 * advert interval + (256 - priority) / 256 * advert interval, which is the timeout for a backup instance not receiving adverts before it makes the decision to transition to master state. The suggests that node 2 is not receiving adverts from node 1.

After node 2 transitions to master state, does node 1 start logging that it is receiving lower priority adverts? This is key to help identifying the cause of the problem.

There are a number of reasons why a backup node does not receive adverts from the current master node:

1. There is some issue with the configuration of the network between the two nodes. It is surprising how often users discover a problem which they did not previously know about, after experiencing this problem.

2. There is a problem with firewall configurations (including iptables/nftables on the two nodes). There needs to be an explicit rule to allow protocol 112 (VRRP protocol) if any traffic is being filtered. Relying on a ctstate ESTABLISHED,RELATED rule will not work reliably.

3. With certain network configurations RP_FILTER must be set to 2, or 0, and definitely not 1. This applies not only to /proc/sys/net/ipv4/conf/eth0/rp_filter (in your case) but also /proc/sys/net/ipv4/conf/all/rp_filter, but keepalived will normally ensure that the settings are correct. If they are subsequently altered by something else, then it can cause problems.

from keepalived.

This appears to be the same issue as #2405 - both have IP address fe80::f816:3eff:fe10:c8c6 and both have a VRRP instance named esc_control_eth_vrrp. Issue #2405 identifies that this is an issue with the setup of OpenStack not passing multicast messages between VMs.

from keepalived.