SONiC switches when connected one to another cannot resolve each other because their ports have the same MAC. We need to provide MAC address distribution based on base MAC from device EEPROM.
root@arc-switch1025:/home/admin# ping 20.0.0.5
PING 20.0.0.5 (20.0.0.5) 56(84) bytes of data.
09:04:57.386041 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
From 20.0.0.4 icmp_seq=1 Destination Host Unreachable
09:04:58.555442 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
09:04:59.554031 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
09:05:00.554003 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
From 20.0.0.4 icmp_seq=2 Destination Host Unreachable
From 20.0.0.4 icmp_seq=3 Destination Host Unreachable
From 20.0.0.4 icmp_seq=4 Destination Host Unreachable
09:05:01.571378 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
09:05:02.570035 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
09:05:03.570071 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
From 20.0.0.4 icmp_seq=5 Destination Host Unreachable
From 20.0.0.4 icmp_seq=6 Destination Host Unreachable
From 20.0.0.4 icmp_seq=7 Destination Host Unreachable
^C
--- 20.0.0.5 ping statistics ---
8 packets transmitted, 0 received, +7 errors, 100% packet loss, time 7031ms
pipe 3
root@arc-switch1025:/home/admin# 09:05:04.587432 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
09:05:05.586050 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
09:05:06.586058 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 28
09:04:53.329642 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:54.329629 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:55.387772 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:55.387809 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:04:56.385577 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:56.385602 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:04:57.385587 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:57.385630 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:04:58.554960 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:58.554989 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:04:59.553578 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:04:59.553621 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:00.553522 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:00.553554 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:01.570977 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:01.571022 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:02.569651 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:02.569698 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:03.569617 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:03.569665 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:04.587050 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:04.587094 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:05.585667 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:05.585712 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
09:05:06.585616 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 20.0.0.5 tell 20.0.0.4, length 46
09:05:06.585666 ARP, Ethernet (len 6), IPv4 (len 4), Reply 20.0.0.5 is-at 00:02:03:04:05:00 (oui Unknown), length 28
Second switch sends ARP reply with same MAC as on first, which cannot be handled correctly.