frrouting / topotests Goto Github PK

this is due to how /proc/sys/kernel/core_pattern is created. Instead should have a core directory per test/node

version_cmp('3.0.2', '3.0.0') == 1
version_cmp('3.0.2', '3.0') == 1

but

version_cmp('3.0.2', '3') == -1

3.0.2 > 3, should return 1

In L3VPN scenario, when I run ip route 4.4.4.4/32 100.1.2.3 vrf test a static route in vrf is defined.
and I run redistribute static in bgp-vrf. Hopefully, static routes are redistributed correctly.
But the problem is when I type no ip route 4.4.4.4/32 100.1.2.3 vrf test, PE router still redistributes the static route!!!!

I believe this problem is for connected routes too.

Hi all. This is my config from one PE side:
router bgp 200 vrf test
bgp router-id 3.3.3.3
!
address-family ipv4 unicast
redistribute static
label vpn export 567
rd vpn export 100:110
rt vpn both 100:1000
export vpn
import vpn
exit-address-family
!
ip route 4.4.4.4/32 100.3.4.4/24 (*)

I can see vpvn4 in other PE and this is ok. The problem is when I delete the static route (*) , I can still see the vpnv4 in other PE which this is not true.
How can I make this correct?

Thank you

e.g., see https://ci1.netdef.org/download/FRR-TOPOPR-T3B/build_logs/FRR-TOPOPR-T3B-150.log
failed to converge after 90 seconds!

build 13-Feb-2018 07:56:13 isis-topo1/test_isis_topo1.py::test_isis_convergence 2018-02-13 07:56:13,875 INFO: loading topology: test_isis_topo1
build 13-Feb-2018 07:56:13 2018-02-13 07:56:13,876 INFO: starting topology: test_isis_topo1
build 13-Feb-2018 07:56:13 2018-02-13 07:56:13,994 INFO: r4: waiting for zebra to start (1 seconds)
build 13-Feb-2018 07:56:15 2018-02-13 07:56:15,240 INFO: r5: waiting for zebra to start (1 seconds)
build 13-Feb-2018 07:56:16 2018-02-13 07:56:16,462 INFO: r1: waiting for zebra to start (1 seconds)
build 13-Feb-2018 07:56:17 2018-02-13 07:56:17,686 INFO: r2: waiting for zebra to start (1 seconds)
build 13-Feb-2018 07:56:18 2018-02-13 07:56:18,926 INFO: r3: waiting for zebra to start (1 seconds)
build 13-Feb-2018 07:56:20 2018-02-13 07:56:20,928 INFO: waiting for ISIS protocol to converge
error 13-Feb-2018 07:57:52 2018-02-13 07:57:52,393 ERROR: assert failed at "test_isis_topo1/test_isis_convergence": Router 'r4' topology mismatch after +91.27 secs

This change:

diff --git a/lib/topotest.py b/lib/topotest.py
index 221910b..1fa4c83 100644
--- a/lib/topotest.py
+++ b/lib/topotest.py
@@ -158,6 +158,9 @@ class Router(Node):
print("LDP Test, but no ldpd compiled or installed")
return "LDP Test, but no ldpd compiled or installed"
kernel_version = re.search(r'([0-9]+.[0-9]+).*', platform.release())
+

•        print("Running Kernel Version: %s" % kernel_version)
  

•        print("Platform Release: %s" % platform.release())
         if kernel_version:
             if float(kernel_version.group(1)) < 4.5:
                 print("LDP Test need Linux Kernel 4.5 minimum")
  

Gives us this output:

Running Kernel Version: <_sre.SRE_Match object at 0x7f25707e97b0>
Platform Release: 4.11.0-041100rc8-generic
LDP Test need Linux Kernel 4.5 minimum
SKIPPED

It was noticed recently that the netns tests are failing too often. After some investigation it seems that the bug lies in i386 VRF with netns.

Here are some samples of failing tests (i386):

https://ci1.netdef.org/browse/FRR-FRRPULLREQ-TOPOI386-2838/artifact
https://ci1.netdef.org/browse/FRR-FRRPULLREQ-TOPOI386-2848/artifact
https://ci1.netdef.org/browse/FRR-FRRPULLREQ-TOPOI386-2845/artifact
https://ci1.netdef.org/browse/FRR-FRRPULLREQ-TOPOI386-2842/artifact

They all fail with the same zebra log errors:
zebra standard output:

2018/03/11 05:12:08 ZEBRA: VRF_GET: (NULL)(1)
2018/03/11 05:12:53 ZEBRA: release_daemon_chunks: Released 0 label chunks
2018/03/11 05:12:53 ZEBRA: client 18 disconnected. 0 system routes removed from the rib
2018/03/11 05:12:53 ZEBRA: release_daemon_chunks: Released 0 label chunks
2018/03/11 05:12:53 ZEBRA: client 17 disconnected. 0 system routes removed from the rib
2018/03/11 05:12:53 ZEBRA: Terminating on signal
2018/03/11 05:12:53 ZEBRA: vrf_terminate: Shutting down vrf subsystem
2018/03/11 05:12:53 ZEBRA: VRF 1 is to be deleted.
2018/03/11 05:12:53 ZEBRA: VRF 1 is to be disabled.
2018/03/11 05:12:53 ZEBRA: VRF 4294967295 is to be deleted.
ZEBRA: Received signal 11 at 1520770373 (si_addr 0x4, PC 0xb7f0af66); aborting...
Program counter: /usr/lib/libfrr.so.0(hash_clean+0x26)[0xb7f0af66]
Backtrace for 15 stack frames:
/usr/lib/libfrr.so.0(zlog_backtrace_sigsafe+0x48)[0xb7f13e58]
/usr/lib/libfrr.so.0(zlog_signal+0x340)[0xb7f14690]
/usr/lib/libfrr.so.0(+0x46ded)[0xb7f29ded]
[0xb7f8bcfc]
/usr/lib/libfrr.so.0(hash_clean+0x26)[0xb7f0af66]
/usr/lib/frr/zebra(zebra_ns_disable+0x2f)[0x4cb4cf]
/usr/lib/frr/zebra(zebra_ns_disabled+0x49)[0x4cb5f9]
/usr/lib/libfrr.so.0(ns_walk_func+0x37)[0xb7f18047]
/usr/lib/frr/zebra(+0x1aa97)[0x4b9a97]
/usr/lib/libfrr.so.0(quagga_sigevent_process+0x50)[0xb7f29ea0]
/usr/lib/libfrr.so.0(thread_fetch+0x608)[0xb7f36348]
/usr/lib/libfrr.so.0(frr_run+0xb6)[0xb7f12726]
/usr/lib/frr/zebra(main+0x318)[0x4ac5f8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf7)[0xb7d13637]
/usr/lib/frr/zebra(+0xdc45)[0x4acc45]
no thread information available

Zebra standard error output:

Hi all.
Would it be possible to add linux kernel configuration here ?
I run these commands before I run l2vpn on frr:
ip link add type bridge
ip link set dev bridge0 up
ip link set dev eth0 up
ip link set dev eth0 master bridge0
ip link add name mpw0 type dummy
ip link set dev mpw0 up
ip link set dev mpw0 master bridge0

https://ci1.netdef.org/browse/FRR-FRR-1065/artifact/TOPOI386/MemoryLeaks/memleak_bgp_l3vpn_to_bgp_vrf.test_bgp_l3vpn_to_bgp_vrf.txt

• note this is similar to #88

Please add IPv6 BGP test?

Is it possible to use some versioning on topotests.
The goal is to be aware of the topotest version currently tested for users.

topotests has features that are added.
speaking with topotests version is nicer than speaking with sha1.

my proposal is to set a tag 1.0 for instance on one of the next commits merged on topotests master.

we could then align with versioning workflow with frr:

• new features : 1.0 => 2.0
• bug fixes : 1.0 => 1.1 ( not for each commit, it is a maintainers decision, I think).

e.g., no error on https://ci1.netdef.org/browse/FRR-FRR-1065/artifact/TOPOI386/MemoryLeaks/memleak_bgp_l3vpn_to_bgp_vrf.test_bgp_l3vpn_to_bgp_vrf.txt

e.g., https://ci1.netdef.org/browse/FRR-FRR-1065/artifact/TOPOI386/MemoryLeaks/memleak_bgp_l3vpn_to_bgp_vrf.test_bgp_l3vpn_to_bgp_vrf.txt resulted from a crash, but no stack is available.

change proposed in
https://github.com/pguibert6WIND/topotests/tree/bgp_vrf_to_bgp_vpn-vmerge1-namingvrf
does the following:

• at least provide the vrf renaming per machine
• then provide a mechanism to test with vrf with netns.

https://ci1.netdef.org/browse/FRR-FRRPULLREQ-TOPOU1604-505/log

Every daemon cores on shutdown but I still get the green checkmark.

Please document that topotests uses SIGBUS to acquire memory allocations at shutdown to avoid others attempting to debug it.

when I type " rt vpn both 52:100", there is an error occur:
vtysh: error reading from bgpd: Success (0)Warning: closing connection to bgpd because of an I/o error!

There is an issue in zebra, that is not appearing with bgp vrf netns test setup.
sounds like NETNS can not be read by zebra.

Zebra is used in the same way for bgp vrf netns test setup.
So I can exclude issues related to user capability ( frr user is used, right?).

Here, I was wondering if there was something else that could cause the problem.

2018/07/13 16:06:06 errors: ZEBRA: Can not enable NS 1: Permission denied!
2018/07/13 16:06:06 warnings: ZEBRA: Can not associate NS 1 with NETNS /run/netns/r3-cust1
2018/07/13 16:06:06 warnings: ZEBRA: NS notify : failed to create NS /run/netns/r3-cust1
2018/07/13 16:06:06 errors: ZEBRA: Can not enable NS 3: Permission denied!
2018/07/13 16:06:06 warnings: ZEBRA: Can not associate NS 3 with NETNS /run/netns/r1-cust1
2018/07/13 16:06:06 warnings: ZEBRA: NS notify : failed to create NS /run/netns/r1-cust1

Any changes to the southbound interface are very hard to test across all supported platforms. It would be super useful to have topotests be able to run across all platforms we currently support.

Work is being done to revamp the southbound interface and changes made are fragile on systems beyond linux because there is no good way to look changes on all systems

In the same order of checkpatch for frr, I think we also need to verify topotest Pull Request, at least with pep8 and pylint. But, running both on my local copy of topotest give very low score.

I discuss this with one of my colleague (PTL for functest at OPNFV). He told me that the topotest directory was not conform to python rules. He recommends me to look at [https://github.com/opnfv/functest] and try to organize topotest in the same way. Once done, we could use tox to automate python style checking same way we are checking C style with checkpath.

Of course, this imply a certain amount of work (not too many said my colleague) to re-factor topotest code according to python rules.

What do you think about this idea ?

e.g., see https://ci1.netdef.org/browse/FRR-TOPOPR-TMB-251/

  assert error_logs == "", "Daemons report errors to StdErr"

E AssertionError: Daemons report errors to StdErr
E assert 'r1 ISISd Std...bugging is on' == ''
E - r1 ISISd StdErr Output:
E - 2017/08/28 12:07:22 warnings: ISIS: ISIS-Spf: TENT is empty SPF-root:robot\r
E - 2017/08/28 12:07:22 warnings: ISIS: ISIS-Spf: TENT is empty SPF-root:robot\r
E - 2017/08/28 12:07:22 warnings: ISIS: ISIS-Spf: TENT is empty SPF-root:robot\r
E - 2017/08/28 12:07:22 warnings: ISIS: ISIS-Spf: TENT is empty SPF-root:robot\r
E - IS-IS Event debugging is on

all-protocol-startup/test_all_protocol_startup.py:276: AssertionError
============== 1 failed, 68 passed, 14 skipped in 701.64 seconds ===============

Write some basic tests for labeled-unicast and ensure that they are working properly for both v4 and v6

I installed topotests on ubuntu 18.04 and am attempting to run the topotests, I am getting a non-trivial number of failure cases based upon what looks like whitespace differences between what the tests expect and what is being displayed.

r1 failed Linux IPv6 Kernel Routing Table Check:
--- actual OSPFv3 IPv6 routing table
+++ expected OSPFv3 IPv6 routing table
@@ -1,13 +1,13 @@
-fc00:1:1:1::/64 dev r1-stubnet  proto XXXX metric 256 pref medium
-fc00:2:2:2::/64 via fe80::__(r2-sw5)__ dev r1-sw5  proto XXXX metric 20 pref medium
-fc00:3:3:3::/64 via fe80::__(r3-sw5)__ dev r1-sw5  proto XXXX metric 20 pref medium
-fc00:4:4:4::/64 via fe80::__(r3-sw5)__ dev r1-sw5  proto XXXX metric 20 pref medium
-fc00:a:a:a::/64 dev r1-sw5  proto XXXX metric 256 pref medium
-fc00:b:b:b::/64 via fe80::__(r3-sw5)__ dev r1-sw5  proto XXXX metric 20 pref medium
-fc00:1111:1111:1111::/64 via fc00:1:1:1::1234 dev r1-stubnet  proto XXXX metric 20 pref medium
-fc00:2222:2222:2222::/64 via fe80::__(r2-sw5)__ dev r1-sw5  proto XXXX metric 20 pref medium
-fc00:3333:3333:3333::/64 via fe80::__(r3-sw5)__ dev r1-sw5  proto XXXX metric 20 pref medium
-fc00:4444:4444:4444::/64 via fe80::__(r3-sw5)__ dev r1-sw5  proto XXXX metric 20 pref medium
-unreachable fe80::/64 dev lo  proto XXXX metric 256 error -101 pref medium
-fe80::/64 dev r1-stubnet  proto XXXX metric 256 pref medium
-fe80::/64 dev r1-sw5  proto XXXX metric 256 pref medium
+fc00:1:1:1::/64 dev r1-stubnet  proto XXXX  metric 256  pref medium
+fc00:2:2:2::/64 via fe80::__(r2-sw5)__ dev r1-sw5  proto XXXX  metric 20  pref medium
+fc00:3:3:3::/64 via fe80::__(r3-sw5)__ dev r1-sw5  proto XXXX  metric 20  pref medium
+fc00:4:4:4::/64 via fe80::__(r3-sw5)__ dev r1-sw5  proto XXXX  metric 20  pref medium
+fc00:a:a:a::/64 dev r1-sw5  proto XXXX  metric 256  pref medium
+fc00:b:b:b::/64 via fe80::__(r3-sw5)__ dev r1-sw5  proto XXXX  metric 20  pref medium
+fc00:1111:1111:1111::/64 via fc00:1:1:1::1234 dev r1-stubnet  proto XXXX  metric 20  pref medium
+fc00:2222:2222:2222::/64 via fe80::__(r2-sw5)__ dev r1-sw5  proto XXXX  metric 20  pref medium
+fc00:3333:3333:3333::/64 via fe80::__(r3-sw5)__ dev r1-sw5  proto XXXX  metric 20  pref medium
+fc00:4444:4444:4444::/64 via fe80::__(r3-sw5)__ dev r1-sw5  proto XXXX  metric 20  pref medium
+unreachable fe80::/64 dev lo  proto XXXX  metric 256  error -101 pref medium
+fe80::/64 dev r1-stubnet  proto XXXX  metric 256  pref medium
+fe80::/64 dev r1-sw5  proto XXXX  metric 256  pref medium```

Is an example of the failure.