skupperproject / skupper-router Goto Github PK

View Code? Open in Web Editor NEW

14.0 14.0 17.0 33.33 MB

An application-layer router for Skupper networks

Home Page: https://skupper.io

License: Apache License 2.0

CMake 0.89% Dockerfile 0.05% Shell 0.62% Python 40.99% HTML 1.86% C 46.11% C++ 9.08% Rust 0.40%

skupper-router's People

Contributors

Stargazers

Watchers

Forkers

ted-ross pwright kgiusti jiridanek ganeshmurthy fgiorgetti bartoval mgoulish grs nluaces ajssmith ekmixon astitcher rafaelvzago chugr ssorj cliffjansen

skupper-router's Issues

Remove deprecated attributes and entities from the skupper-router schema

There are many deprecated attributes and entities in the skupper-router schema. Remove these deprecated items along with any tests that test these.

Remove link-routing capability

Remove the link-routing support. This will greatly simplify testing and future development of new routing features. Most of the unique and valuable benefits of link-routing can be achieved using message streaming and non-terminal delivery dispositions.

Remove Fallback destination feature

Update CMake: 2.8.12 -> 3.20

There should not be any large gotchas, with maybe the exception of Policy CMP0065. CMake used to compile all binaries (under GCC) with -rdynamic by default, but according to this new policy it removed the flag. Dispatch needs the flag so it can dlopen() its own binary. Any tests that do router initializations also need the flag.

The solution is to either set ENABLE_EXPORTS for each target that needs it, or to set CMAKE_ENABLE_EXPORTS to get the old behavior.

Remove libwebsockets-devel from the final docker image

According to discussion at #44 (comment), the -devel version can likely be replaced with nondevel version, which will make the resulting image slightly smaller.

Enable CI via Github actions

Router crash in system_tests_http2.Http2TestInteriorEdgeRouter - use after free

Not reliably reproducible.

Build Info:
Os: Fedora 34
Skupper-router: commit b2a1748
Skupper-build: Debug (no runtime checks)
Proton: commit ec535e772e02bd67a34a90d301c450758f53833e
Proton build: RelWithDebInfo (no runtime checks)

$ ctest -V -j8
....
58: ======================================================================                                                                        
58: ERROR: tearDownClass (system_tests_http2.Http2TestInteriorEdgeRouter)                                                                         
58: ----------------------------------------------------------------------                                                                        
58: Traceback (most recent call last):                                                                                                            
58:   File "/home/kgiusti/work/skupper/skupper-router/tests/system_test.py", line 851, in tearDownClass                                           
58:     cls.tester.teardown()                                                                                                                     
58:   File "/home/kgiusti/work/skupper/skupper-router/tests/system_test.py", line 794, in teardown                                                
58:     raise RuntimeError("Errors during teardown: \n\n%s" % "\n\n".join([str(e) for e in errors]))                                              
58: RuntimeError: Errors during teardown:                                                                                                         
58:                                                                                                                                               
58: Process 380029 error: exit code -11, expected 0                                                                                               
58: qdrouterd -c edge-router.conf -I /home/kgiusti/work/skupper/skupper-router/python                                                             
58: /home/kgiusti/work/skupper/skupper-router/BUILD/tests/system_test.dir/system_tests_http2/Http2TestInteriorEdgeRouter/setUpClass/edge-router-5\
2.cmd

$ gdb
...
  Command Line: qdrouterd -c edge-router.conf -I /home/kgiusti/work/skupper/skupper-router/python  
...
Program terminated with signal SIGSEGV, Segmentation fault.                                                                                       
#0  0x000000000047b105 in qdr_delivery_message (delivery=0x9999999999999999)                                                                      
    at /home/kgiusti/work/skupper/skupper-router/src/router_core/delivery.c:176                                                                   
176         return delivery->msg;                                                                                                                 
[Current thread is 1 (Thread 0x7f2e7302c640 (LWP 380032))]                                                                                        
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.33-21.fc34.x86_64 libdb-5.3.28-49.fc34.x86_64 libffi-3.1-28.fc34.x86_64 libgcc-11\
.2.1-9.fc34.x86_64 libselinux-3.2-1.fc34.x86_64 libxcrypt-4.4.27-2.fc34.x86_64 pcre2-10.36-4.fc34.x86_64 python3.10-3.10.2-1.fc34.x86_64          
(gdb) bt                                                                                                                                          
#0  0x000000000047b105 in qdr_delivery_message (delivery=0x9999999999999999)                                                                      
    at /home/kgiusti/work/skupper/skupper-router/src/router_core/delivery.c:176                                                                   
#1  0x00000000004ae9a0 in read_data_callback (session=0x12da650, stream_id=19,                                                                    
    buf=0x12dc46a "B\003PUT\004\206b\237F\252S\377\206\306\305\304\017\r\002\062\060\300", length=16384, data_flags=0x7f2e7302ac40,               
    source=0x13ed5d0, user_data=0x126a910) at /home/kgiusti/work/skupper/skupper-router/src/adaptors/http2/http2_adaptor.c:1178                   
#2  0x00007f2e821df38b in nghttp2_session_pack_data (stream=0x17e3730, aux_data=0x13ed5d0, frame=0x13ed570, datamax=16384, bufs=0x12da7a8,        
    session=0x12da650) at /usr/src/debug/nghttp2-1.43.0-2.fc34.x86_64/lib/nghttp2_session.c:7173                                                  
#3  session_prep_frame (item=0x13ed570, session=0x12da650) at /usr/src/debug/nghttp2-1.43.0-2.fc34.x86_64/lib/nghttp2_session.c:2047              
#4  nghttp2_session_mem_send_internal (session=session@entry=0x12da650, data_ptr=data_ptr@entry=0x7f2e7302ac90, fast_cb=fast_cb@entry=0)          
    at /usr/src/debug/nghttp2-1.43.0-2.fc34.x86_64/lib/nghttp2_session.c:2932                                                                     
#5  0x00007f2e821e06a9 in nghttp2_session_send (session=0x12da650) at /usr/src/debug/nghttp2-1.43.0-2.fc34.x86_64/lib/nghttp2_session.c:3254      
#6  0x00000000004b04b8 in qdr_http_delivery_update (context=0x11f8b40, dlv=0x15cf790, disp=38, settled=true)                                      
    at /home/kgiusti/work/skupper/skupper-router/src/adaptors/http2/http2_adaptor.c:1601                                                          
#7  0x0000000000471633 in qdr_connection_process (conn=0x127f250)                                                                                 
    at /home/kgiusti/work/skupper/skupper-router/src/router_core/connections.c:390                                                                
#8  0x00000000004b4ac4 in handle_connection_event (e=0x11b2fe0, qd_server=0x1103fe0, context=0x126a910)                                           
    at /home/kgiusti/work/skupper/skupper-router/src/adaptors/http2/http2_adaptor.c:2545                                                          
#9  0x00000000004a294c in handle_event_with_context (e=0x11b2fe0, qd_server=0x1103fe0, context=0x126a910)                                         
    at /home/kgiusti/work/skupper/skupper-router/src/server.c:777                                                                                 
#10 0x00000000004a2989 in do_handle_raw_connection_event (e=0x11b2fe0, qd_server=0x1103fe0)                                                       
    at /home/kgiusti/work/skupper/skupper-router/src/server.c:783                                                                                 
#11 0x00000000004a3813 in handle (qd_server=0x1103fe0, e=0x11b2fe0, pn_conn=0x0, ctx=0x0)                                                         
    at /home/kgiusti/work/skupper/skupper-router/src/server.c:1060                                                                                
#12 0x00000000004a3921 in thread_run (arg=0x1103fe0) at /home/kgiusti/work/skupper/skupper-router/src/server.c:1092                               
#13 0x00000000004660d3 in _thread_init (arg=0x119f790) at /home/kgiusti/work/skupper/skupper-router/src/posix/threading.c:172                     
#14 0x00007f2e825ef2a5 in start_thread () from /lib64/libpthread.so.0                                                                             
#15 0x00007f2e82104323 in clone () from /lib64/libc.so.6 

... 
(gdb) list                                                                                                                                        
1173                          nghttp2_data_source *source,                                                                                        
1174                          void *user_data)                                                                                                    
1175    {                                                                                                                                         
1176        qdr_http2_connection_t *conn = (qdr_http2_connection_t *)user_data;                                                                   
1177        qdr_http2_stream_data_t *stream_data = (qdr_http2_stream_data_t *)source->ptr;                                                        
1178 -->  qd_message_t *message = qdr_delivery_message(stream_data->out_dlv);                                                                   
1179        qd_message_depth_status_t status = qd_message_check_depth(message, QD_DEPTH_BODY);                                                    
1180                                                                                                                                              
1181        // This flag tells nghttp2 that the data is not being copied into the buffer supplied by nghttp2 (uint8_t *buf).                      
1182        *data_flags |= NGHTTP2_DATA_FLAG_NO_COPY;                                                                                             
(gdb) p *conn                                                                                                                                     
$3 = {context = {context = 0x126a910, handler = 0x4b44b6 <handle_connection_event>}, qdr_conn = 0x127f250, pn_raw_conn = 0x12bf690,               
  read_buffers = {{context = 0, bytes = 0x0, capacity = 0, size = 0, offset = 0}, {context = 0, bytes = 0x0, capacity = 0, size = 0,              
      offset = 0}, {context = 0, bytes = 0x0, capacity = 0, size = 0, offset = 0}, {context = 0, bytes = 0x0, capacity = 0, size = 0,             
      offset = 0}}, activate_timer = 0x11b47d0, ping_timer = 0x0, config = 0x125f610, server = 0x1103fe0, conn_id = 1, remote_address = 0x0,      
  stream_dispatcher = 0x1292bd0, stream_dispatcher_stream_data = 0x12a7ed0, stream_dispatcher_id = 1, data_prd = {source = {fd = 22196560,        
      ptr = 0x152b150}, read_callback = 0x4ae95e <read_data_callback>}, granted_read_buffs = {head = 0x131ee90, tail = 0x13686d0,                 
    scratch = 0x0, size = 16}, prev_ping = 0, last_pn_raw_conn_read = 0, buffs = {head = 0x0, tail = 0x0, scratch = 0x13500d0, size = 0},         
  session = 0x12da650, streams = {head = 0x152b390, tail = 0x152b390, scratch = 0x152b390, size = 1}, connection_established = true,              
  grant_initial_buffers = false, ingress = false, timer_scheduled = false, client_magic_sent = true, woken_by_ping = false,                       
  first_pinged = false, delete_egress_connections = false, goaway_received = false, raw_closed_read = 0, raw_closed_write = 0,                    
  q2_blocked = false, q2_restart = 0, delay_buffer_write = 0, prev = 0x0, next = 0x0}                                                             
(gdb) p source                                                                                                                                    
$4 = (nghttp2_data_source *) 0x13ed5d0                                                                                                            
(gdb) p *source                                                                                                                                   
$5 = {fd = 22196560, ptr = 0x152b150}                                                                                                             
(gdb) p stream_data                                                                                                                               
$6 = (qdr_http2_stream_data_t *) 0x152b150                                                                                                        
(gdb) p *stream_data                                                                                                                              
$7 = {conn = 0x9999999999999999, context = 0x9999999999999999,                                                                                    
  reply_to = 0x9999999999999999 <error: Cannot access memory at address 0x9999999999999999>,                                                      
  remote_site = 0x9999999999999999 <error: Cannot access memory at address 0x9999999999999999>,                                                   
  method = 0x9999999999999999 <error: Cannot access memory at address 0x9999999999999999>,                                                        
  request_status = 0x9999999999999999 <error: Cannot access memory at address 0x9999999999999999>, in_dlv = 0x9999999999999999,                   
  out_dlv = 0x9999999999999999, incoming_id = 11068046444225730969, outgoing_id = 11068046444225730969,                                           
  out_dlv_local_disposition = 11068046444225730969, in_link = 0x9999999999999999, out_link = 0x9999999999999999, message = 0x9999999999999999,    
  app_properties = 0x9999999999999999, footer_properties = 0x9999999999999999, body_buffers = {head = 0x9999999999999999,                         
    tail = 0x9999999999999999, scratch = 0x9999999999999999, size = 11068046444225730969}, curr_stream_data = 0x9999999999999999,                 
  curr_stream_data_iter = 0x9999999999999999, next_stream_data = 0x9999999999999999, footer_stream_data = 0x9999999999999999,                     
  footer_stream_data_iter = 0x9999999999999999, prev = 0x9999999999999999, next = 0x9999999999999999, curr_stream_data_result = 2576980377,       
  next_stream_data_result = 2576980377, payload_handled = -1717986919, in_link_credit = -1717986919, stream_id = -1717986919,                     
  status = (QD_STREAM_HALF_CLOSED | unknown: 0x99999998), entire_footer_arrived = 153, entire_header_arrived = 153, out_msg_header_sent = 153,    
  out_msg_body_sent = 153, use_footer_properties = 153, full_payload_handled = 153, out_msg_has_body = 153, out_msg_data_flag_eof = 153,          
  out_msg_has_footer = 153, out_msg_send_complete = 153, disp_updated = 153, disp_applied = 153, header_and_props_composed = 153,                 
  stream_force_closed = 153, in_dlv_decrefed = 153, out_dlv_decrefed = 153, body_data_added_to_msg = 153, bytes_in = -1717986919,                 
  bytes_out = -1717986919, start = -7378697629483820647, stop = -7378697629483820647}                                                             
(gdb) quit

$ tail ./tests/system_test.dir/system_tests_http2/Http2TestInteriorEdgeRouter/setUpClass/edge-router.log      
2022-02-17 10:35:21.924215 -0500 ROUTER_CORE (debug) [C2][L15][D77] Delivery outcome : is released (0x26) (/home/kgiusti/work/skupper/skupper-rou\
ter/src/router_core/delivery.c:392)                                                                                                               
2022-02-17 10:35:21.924295 -0500 HTTP_ADAPTOR (trace) [C1] Activation triggered, calling pn_raw_connection_wake() (/home/kgiusti/work/skupper/sku\
pper-router/src/adaptors/http2/http2_adaptor.c:1705)                                                                                              
2022-02-17 10:35:21.924361 -0500 ROUTER_CORE (trace) Core action 'link_flow' (/home/kgiusti/work/skupper/skupper-router/src/router_core/router_co\
re_thread.c:235)                                                                                                                                  
2022-02-17 10:35:21.924383 -0500 HTTP_ADAPTOR (trace) [C1] PN_RAW_CONNECTION_WAKE Wake-up (/home/kgiusti/work/skupper/skupper-router/src/adaptors\
/http2/http2_adaptor.c:2538)                                                                                                                      
2022-02-17 10:35:21.924410 -0500 ROUTER_CORE (trace) Core action 'link_detach' (/home/kgiusti/work/skupper/skupper-router/src/router_core/router_\
core_thread.c:235)                                                                                                                                
2022-02-17 10:35:21.924444 -0500 ROUTER_CORE (info) [C4][L7] Link lost: del=1 presett=0 psdrop=0 acc=1 rej=0 rel=0 mod=0 delay1=0 delay10=0 block\
ed=no (/home/kgiusti/work/skupper/skupper-router/src/router_core/connections.c:1090)                                                              
2022-02-17 10:35:21.924461 -0500 ROUTER_CORE (trace) Core action 'link_detach' (/home/kgiusti/work/skupper/skupper-router/src/router_core/router_\
core_thread.c:235)                                                                                                                                
2022-02-17 10:35:21.924479 -0500 ROUTER_CORE (info) [C4][L8] Link lost: del=1 presett=1 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 block\
ed=no (/home/kgiusti/work/skupper/skupper-router/src/router_core/connections.c:1090)                                                              
2022-02-17 10:35:21.924499 -0500 ROUTER_CORE (trace) Core action 'connection_closed' (/home/kgiusti/work/skupper/skupper-router/src/router_core/r\
outer_core_thread.c:235)                                                                                                                          
2022-02-17 10:35:21.924515 -0500 ROUTER_CORE (info) [C4] Connection Closed (/home/kgiusti/work/skupper/skupper-router/src/router_core/connections\
.c:1562)                                                                                                                                          
[kgiusti@localhost BUILD (ISSUE-79)]$

Remove sunOS code in CMakeLists and in the router

Best addressed after resolving #13 first.

message.c:1066: void qd_message_free(qd_message_t*): Assertion `DEQ_IS_EMPTY(msg->stream_data_list)' failed

I've very occasionally seen a crash on the following assert, when running the scenario from https://issues.apache.org/jira/browse/DISPATCH-2192. I would like to investigate further and maybe come up with reliable reproduction steps. As far as I can tell, I was doing, in some order, with some repetitions

iperf3 -c 127.0.0.1 -p 8000
iperf3 -c 127.0.0.1 -p 8000 -P 100 -n 9999999999999, ^C
^C'd a router, then started it again

qdrouterd: /home/jdanek/repos/skupper-router/src/message.c:1066: void qd_message_free(qd_message_t*): Assertion `DEQ_IS_EMPTY(msg->stream_data_list)' failed.

There was the same problem in qpid-dispatch and there it also was hard to reliably reproduce, https://issues.apache.org/jira/browse/DISPATCH-2165.

Fallback destination deprecation and removal

AMQP Transactions - local coordinator deprecation and removal

Multi-tenancy deprecation and removal

Generalize the MAU data format in the router protocol

In order to support end-to-end flow control and other more advanced routing features, the MAU (Mobile Address Update) message format needs to be enhanced such that data can be carried along with addresses. Formerly, the existence or non-existence of an address was the extent of the data encoded in this message.

This is a breaking (non backward compatible) change to the protocol and should be completed prior to the first official release.

Advertise correct product Id in Open performative

Build and publish skupper-router image

Remove console code and build tooling

system_tests_one_router fails in test_30_presettled_overflow test

 self = <system_tests_one_router.OneRouterTest testMethod=test_30_presettled_overflow>

    def test_30_presettled_overflow(self):
        test = PresettledOverflowTest(self.address)
        test.run()
>       self.assertIsNone(test.error)
       AssertionError: 'No deliveries were dropped' is not None

https://github.com/skupperproject/skupper-router/runs/5186887345?check_suite_focus=true#step:27:117

Remove downgrade of the python version in GHA configuration

Use the default system version of python instead.

Need to remove this step from the workflow configuration.

Port DISPATCH-2309 from Qpid Dispatch

Authorization service plug-in deprecation and removal

AutoLink deprecation and removal

Create CONTRIBUTING.md

Move per message router annotations to a custom message header

POC: determine code and performance impact.

Keeping in sync with apache/qpid-dispatch

Things I'll need to get merged here

And that's all

Port DISPATCH-2317 from Qpid Dispatch

system_tests_http2 failing with error in test_post_upload_large_image_jpg

/home/runner/work/skupper-router/skupper-router/skupper-router/tests/system_tests_http2.py:168: AssertionError
____ Http2TestEdgeToEdgeViaInteriorRouter.test_post_upload_large_image_jpg _____

self = <system_tests_http2.Http2TestEdgeToEdgeViaInteriorRouter testMethod=test_post_upload_large_image_jpg>

    @unittest.skipIf(skip_test(), "Python 3.7 or greater, Quart 0.13.0 or greater and curl needed to run http2 tests")
    @unittest.skipIf(skip_test_in_ci('QPID_SYSTEM_TEST_SKIP_HTTP2_LARGE_IMAGE_UPLOAD_TEST'), skip_reason)
    def test_post_upload_large_image_jpg(self):
        # curl  -X POST -H "Content-Type: multipart/form-data"  -F "data=@/home/gmurthy/opensource/test.jpg"
        # http://127.0.0.1:9000/upload --http2-prior-knowledge
        address = self.router_qdra.http_addresses[0] + "/upload"
        out = self.run_curl(address, args=['-X', 'POST', '-H', 'Content-Type: multipart/form-data',
                                           '-F', 'data=@' + image_file('test.jpg')])
>       self.assertIn('Success', out)
E       AssertionError: 'Success' not found in '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>408 Request Timeout</title>\n<h1>Request Timeout</h1>\n<p>The server closed the network connection because the browser didn&#x27;t finish the request within the specified time.</p>\n'

https://github.com/skupperproject/skupper-router/runs/5350854560?check_suite_focus=true#step:27:4354

Image build - python3 command not found

Observed in current skupper-router image

/home/qdrouterd/bin/configure.sh: line 56: python3: command not found

Exchange/bindings legacy code removal

Port DISPATCH-2132

DISPATCH-2132: atomic access to core uptime counter

system_tests_delivery_counts failing

44: =================================== FAILURES ===================================
44: __________ OneRouterLinkCountersTest.test_06_large_message_released ___________
44: 44: self = <system_tests_delivery_counts.OneRouterLinkCountersTest testMethod=test_06_large_message_released>
44: 
44:     def test_06_large_message_released(self):
44: >       self.verify_released(True)
44: 44: /home/runner/work/skupper-router/skupper-router/skupper-router/tests/system_tests_delivery_counts.py:1198: 
44: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
44: /home/runner/work/skupper-router/skupper-router/skupper-router/tests/system_tests_delivery_counts.py:1059: in verify_released
44:     self.assertEqual(test.receiver_stats['releasedCount'], self.COUNT)
44: E   AssertionError: 29 != 40
44: - generated xml file: /__w/skupper-router/skupper-router/skupper-router/build/tests/junitxmls/system_tests_delivery_counts.xml -
44: =========================== short test summary info ============================
44: FAILED ::OneRouterLinkCountersTest::test_06_large_message_released - Assertio...
44: ======================== 1 failed, 26 passed in 59.43s =========================
23/33 Test #44: system_tests_delivery_counts ...................***Failed   59.76 sec
============================= test session starts ==============================

https://github.com/ganeshmurthy/skupper-router/runs/5109959880?check_suite_focus=true#step:27:1023

Remove Non-linux build targets in travis CI

osx

Port DISPATCH-2308

https://issues.apache.org/jira/browse/DISPATCH-2308

Q2 tests in system_tests_http2 must check for equal number of blocked and unblocked q2 messages

Currently test_q2_block_unblock does the following

self.assertGreater(num_blocked, 0)
self.assertGreater(num_unblocked, 0)

It should also do the following -

self.assertGreaterEqual(num_unblocked, num_blocked)

Make CI builds (especially skupper-router image) more production-like

This is a follow up to (my) comments on #44. This ticket suggests to consider doing (and eventually do) one or more of the following

Create PyCharm / IntelliJ log highlighting configuration

Besides these IDEs, or whatever IDE people use that supports the advanced log viewing features, defining these patterns might also make sense for log exporters or log collectors for production deployments.

TSAN race on setting the presettled flag

18: E           WARNING: ThreadSanitizer: data race (pid=1997)
18: E             Read of size 1 at 0x7b44000277d5 by thread T1:
18: E               #0 qdr_delivery_continue_CT /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/delivery.c:1108 (qdrouterd+0x4945e2)
18: E               #1 router_core_thread /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/router_core_thread.c:236 (qdrouterd+0x4a2e5a)
18: E               #2 _thread_init /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:172 (qdrouterd+0x47a2fd)
18: E           
18: E             Previous write of size 1 at 0x7b44000277d5 by thread T3:
18: E               #0 qdr_delivery_set_presettled /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/delivery.c:136 (qdrouterd+0x4924d5)
18: E               #1 AMQP_rx_handler /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_node.c:526 (qdrouterd+0x4baa28)
18: E               #2 deferred_AMQP_rx_handler /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_node.c:882 (qdrouterd+0x4bb142)
18: E               #3 invoke_deferred_calls /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:764 (qdrouterd+0x4bd1db)
18: E               #4 invoke_deferred_calls /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:753 (qdrouterd+0x4c0466)
18: E               #5 handle /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1017 (qdrouterd+0x4c0466)
18: E               #6 thread_run /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1092 (qdrouterd+0x4c21c4)
18: E               #7 _thread_init /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:172 (qdrouterd+0x47a2fd)
18: E           
18: E             Location is heap block of size 320 at 0x7b4400027740 allocated by thread T4:
18: E               #0 posix_memalign <null> (libtsan.so.0+0x32a23)
18: E               #1 qd_alloc /home/runner/work/skupper-router/skupper-router/skupper-router/src/alloc_pool.c:391 (qdrouterd+0x447b69)
18: E               #2 new_qdr_delivery_t /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/delivery.c:24 (qdrouterd+0x491f4c)
18: E               #3 qdr_link_deliver /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/transfer.c:63 (qdrouterd+0x4a8305)
18: E               #4 AMQP_rx_handler /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_node.c:837 (qdrouterd+0x4bab23)
18: E               #5 AMQP_link_detach_handler /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_node.c:1089 (qdrouterd+0x4bb0ba)
18: E               #6 AMQP_link_detach_handler /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_node.c:1062 (qdrouterd+0x4bb0ba)
18: E               #7 close_links /home/runner/work/skupper-router/skupper-router/skupper-router/src/container.c:328 (qdrouterd+0x458349)
18: E               #8 qd_container_handle_event /home/runner/work/skupper-router/skupper-router/skupper-router/src/container.c:520 (qdrouterd+0x45b234)
18: E               #9 handle /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1067 (qdrouterd+0x4c048e)
18: E               #10 thread_run /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1092 (qdrouterd+0x4c2144)
18: E               #11 _thread_init /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:172 (qdrouterd+0x47a2fd)
18: E           
18: E             Thread T1 (tid=2004, running) created by main thread at:
18: E               #0 pthread_create <null> (libtsan.so.0+0x5bef5)
18: E               #1 sys_thread /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:181 (qdrouterd+0x47a78c)
18: E               #2 qdr_core /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/router_core.c:121 (qdrouterd+0x49d920)
18: E               #3 qd_router_setup_late /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_node.c:2078 (qdrouterd+0x4bb8a8)
18: E               #4 ffi_call_unix64 <null> (libffi.so.6+0x6c03)
18: E               #5 main_process /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:92 (qdrouterd+0x423c3c)
18: E               #6 main /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:359 (qdrouterd+0x42315c)
18: E           
18: E             Thread T3 (tid=2007, running) created by main thread at:
18: E               #0 pthread_create <null> (libtsan.so.0+0x5bef5)
18: E               #1 sys_thread /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:181 (qdrouterd+0x47a78c)
18: E               #2 qd_server_run /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1484 (qdrouterd+0x4c2d7c)
18: E               #3 main_process /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:105 (qdrouterd+0x423c9c)
18: E               #4 main /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:359 (qdrouterd+0x42315c)
18: E           
18: E             Thread T4 (tid=2009, running) created by main thread at:
18: E               #0 pthread_create <null> (libtsan.so.0+0x5bef5)
18: E               #1 sys_thread /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:181 (qdrouterd+0x47a78c)
18: E               #2 qd_server_run /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1484 (qdrouterd+0x4c2d7c)
18: E               #3 main_process /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:105 (qdrouterd+0x423c9c)
18: E               #4 main /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:359 (qdrouterd+0x42315c)
18: E           
18: E           SUMMARY: ThreadSanitizer: data race /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/delivery.c:1108 in qdr_delivery_continue_CT
18: E           ==================
18: E           ThreadSanitizer: reported 1 warnings

https://github.com/skupperproject/skupper-router/runs/5186886991?check_suite_focus=true#step:27:894

Http1AdaptorQ2Standalone backpressure tests, block_ct != unblock_ct failure

See https://issues.apache.org/jira/browse/DISPATCH-2066

Phased Address deprecation and removal

Flow-style logging for the protocol adaptors

A separate logging/tracing facility is needed for the protocol adaptors to serve users that are more accustomed to network/firewall/router logs and are not concerned with AMQP/messaging/link-protocol details.

This feature will make the use of the protocol adaptors more observable and debuggable to network-familiar developers and operators.

🐞 Stuck delivery reported in system_tests_tcp_adaptor.py TcpAdaptorStuckDeliveryTest::test_01_ignore_stuck_deliveries

https://github.com/jiridanek/skupper-router/runs/5026451378?check_suite_focus=true#step:27:2618

TCPStuckDeliveryTest_I1.log

72: E                   AssertionError: 'Stuck delivery: At least one delivery' unexpectedly found in '2022-02-01 19:03:49.081570 +0000 ROUTER_CORE (info) [C2][L10] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 3 seconds (/home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/modules/stuck_delivery_detection/delivery_tracker.c:60)\n' : Stuck deliveries should not be logged!
72: 
72: /home/runner/work/skupper-router/skupper-router/skupper-router/tests/system_tests_tcp_adaptor.py:1119: AssertionError
72: - generated xml file: /__w/skupper-router/skupper-router/skupper-router/build/tests/junitxmls/system_tests_tcp_adaptor.xml -
72: =========================== short test summary info ============================
72: FAILED ::TcpAdaptorStuckDeliveryTest::test_01_ignore_stuck_deliveries - Asser...
72: =================== 1 failed, 9 passed, 2 skipped in 49.22s ====================
35/37 Test #72: system_tests_tcp_adaptor ..........................***Failed   49.43 sec

This is something that does not appear to be previously logged in Dispatch Apache Jira.

AddressSanitizer: use-after-poison in qd_message_extend: `qd_message_content_t *content = MSG_CONTENT(msg);`

Configure the router as in system_tests_http2.py::Http2TestOneStandaloneRouter::test_post_upload_large_image_jpg, then repeatedly run

curl --http2-prior-knowledge http://127.0.0.1:21581/upload -X POST -H 'Content-Type: multipart/form-data' -F data=@/home/jdanek/repos/skupper-router/tests/images/test.jpg

=================================================================
==909120==ERROR: AddressSanitizer: use-after-poison on address 0x614000047ef0 at pc 0x000000530e58 bp 0x7f360fb20900 sp 0x7f360fb208f8
READ of size 8 at 0x614000047ef0 thread T3
    #0 0x530e57 in qd_message_extend /home/jdanek/repos/skupper-router/src/message.c:2399
    #1 0x53c318 in qd_message_stream_data_append /home/jdanek/repos/skupper-router/src/message.c:3028
    #2 0x68ab09 in on_data_chunk_recv_callback /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:560
    #3 0x7f3625a6c753 in nghttp2_session_mem_recv (/lib64/libnghttp2.so.14+0xe753)
    #4 0x6946cb in handle_incoming_http /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2133
    #5 0x6b1852 in handle_connection_event /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2542
    #6 0x662fd4 in handle_event_with_context /home/jdanek/repos/skupper-router/src/server.c:780
    #7 0x662fd4 in do_handle_raw_connection_event /home/jdanek/repos/skupper-router/src/server.c:786
    #8 0x662fd4 in handle /home/jdanek/repos/skupper-router/src/server.c:1063
    #9 0x66a6a5 in thread_run /home/jdanek/repos/skupper-router/src/server.c:1095
    #10 0x7f3624f6fb19 in start_thread (/lib64/libc.so.6+0x8db19)
    #11 0x7f3624ff464f in __GI___clone3 (/lib64/libc.so.6+0x11264f)

0x614000047ef0 is located 176 bytes inside of 384-byte region [0x614000047e40,0x614000047fc0)
allocated by thread T4 here:
    #0 0x7f3625f654fc in __interceptor_posix_memalign (/lib64/libasan.so.6+0xaf4fc)
    #1 0x4bb49d in qd_alloc /home/jdanek/repos/skupper-router/src/alloc_pool.c:391
    #2 0x522268 in new_qd_message_t /home/jdanek/repos/skupper-router/src/message.c:93
    #3 0x522268 in qd_message /home/jdanek/repos/skupper-router/src/message.c:1011
    #4 0x68329c in create_http2_stream_data /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:478
    #5 0x6ae964 in qdr_http_deliver /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2032
    #6 0x611d26 in qdr_link_process_deliveries /home/jdanek/repos/skupper-router/src/router_core/transfer.c:194
    #7 0x58a621 in qdr_connection_process /home/jdanek/repos/skupper-router/src/router_core/connections.c:401
    #8 0x6affbe in handle_connection_event /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2545
    #9 0x662fd4 in handle_event_with_context /home/jdanek/repos/skupper-router/src/server.c:780
    #10 0x662fd4 in do_handle_raw_connection_event /home/jdanek/repos/skupper-router/src/server.c:786
    #11 0x662fd4 in handle /home/jdanek/repos/skupper-router/src/server.c:1063
    #12 0x66a6a5 in thread_run /home/jdanek/repos/skupper-router/src/server.c:1095
    #13 0x7f3624f6fb19 in start_thread (/lib64/libc.so.6+0x8db19)

Thread T3 created by T0 here:
    #0 0x7f3625f0c866 in pthread_create (/lib64/libasan.so.6+0x56866)
    #1 0x55f155 in sys_thread /home/jdanek/repos/skupper-router/src/posix/threading.c:181
    #2 0x66cc1c in qd_server_run /home/jdanek/repos/skupper-router/src/server.c:1489
    #3 0x6b633d in main_process /home/jdanek/repos/skupper-router/router/src/main.c:105
    #4 0x432332 in main /home/jdanek/repos/skupper-router/router/src/main.c:359
    #5 0x7f3624f0f55f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f)

Thread T4 created by T0 here:
    #0 0x7f3625f0c866 in pthread_create (/lib64/libasan.so.6+0x56866)
    #1 0x55f155 in sys_thread /home/jdanek/repos/skupper-router/src/posix/threading.c:181
    #2 0x66cc1c in qd_server_run /home/jdanek/repos/skupper-router/src/server.c:1489
    #3 0x6b633d in main_process /home/jdanek/repos/skupper-router/router/src/main.c:105
    #4 0x432332 in main /home/jdanek/repos/skupper-router/router/src/main.c:359
    #5 0x7f3624f0f55f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f)

SUMMARY: AddressSanitizer: use-after-poison /home/jdanek/repos/skupper-router/src/message.c:2399 in qd_message_extend
Shadow bytes around the buggy address:
  0x0c2880000f80: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c2880000f90: 00 00 00 00 00 00 00 00 00 00 f7 f7 f7 f7 f7 f7
  0x0c2880000fa0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0c2880000fb0: f7 f7 f7 f7 00 00 00 00 fa fa fa fa fa fa fa fa
  0x0c2880000fc0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
=>0x0c2880000fd0: 00 00 00 00 00 00 00 00 00 00 f7 f7 f7 f7[f7]f7
  0x0c2880000fe0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0c2880000ff0: f7 f7 f7 f7 00 00 00 00 fa fa fa fa fa fa fa fa
  0x0c2880001000: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2880001010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2880001020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==909120==ABORTING

From the same test, I've also seen the following. See last leak for addr2line decoded trace

addr2line --exe router/qdrouterd --pretty-print 0x<addr>

alloc.c: Items of type 'qd_http2_buffer_t' remain allocated at shutdown: 10
Leak: 2022-02-27 14:54:31.996089 +0100 type: qd_http2_buffer_t address: 0x62900007d290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]

Leak: 2022-02-27 14:54:31.996285 +0100 type: qd_http2_buffer_t address: 0x629000082290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]

Leak: 2022-02-27 14:54:31.996481 +0100 type: qd_http2_buffer_t address: 0x629000168290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]

Leak: 2022-02-27 14:54:31.996670 +0100 type: qd_http2_buffer_t address: 0x6290000af290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]

Leak: 2022-02-27 14:54:31.996820 +0100 type: qd_http2_buffer_t address: 0x6290000e1290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]

Leak: 2022-02-27 14:54:31.996987 +0100 type: qd_http2_buffer_t address: 0x6290002e9290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]

Leak: 2022-02-27 14:54:31.997152 +0100 type: qd_http2_buffer_t address: 0x629000325290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]

Leak: 2022-02-27 14:54:31.997321 +0100 type: qd_http2_buffer_t address: 0x629000069290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]

Leak: 2022-02-27 14:54:31.997487 +0100 type: qd_http2_buffer_t address: 0x6290001ef290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]

Leak: 2022-02-27 14:54:31.997619 +0100 type: qd_http2_buffer_t address: 0x6290001e0290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce] /home/jdanek/repos/skupper-router/src/alloc_pool.c:349
qdrouterd() [0x697522] /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:51
qdrouterd() [0x698111] /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:154
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca] /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2173
qdrouterd() [0x6b036e] /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2555
qdrouterd() [0x662fd5] /home/jdanek/repos/skupper-router/src/server.c:780
qdrouterd() [0x66a6a6] /home/jdanek/repos/skupper-router/src/server.c:1095

system_tests_open_properties test failure in test_02_verify_connector_properties

::OpenPropertiesConfigTest::test_02_verify_connector_properties FAILED
::OpenPropertiesQdManageTest::test_01_verify_listener_properties PASSED
::OpenPropertiesQdManageTest::test_02_verify_connector_properties PASSED
::OpenPropertiesBadConfigTest::test_01_invalid_properties_check PASSED
::OpenPropertiesBadConfigTest::test_02_invalid_role_check PASSEDException in thread Thread-7:
Traceback (most recent call last):
  File "/usr/lib64/python3.6/threading.py", line 937, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.6/threading.py", line 885, in run
    self._target(*self._args, **self._kwargs)
  File "/home/runner/work/skupper-router/skupper-router/skupper-router/tests/test_broker.py", line 113, in _main
    while self._container.process():
  File "/usr/local/lib64/python3.6/site-packages/proton/_reactor.py", line 257, in process
    event.dispatch(handler)
  File "/usr/local/lib64/python3.6/site-packages/proton/_events.py", line 164, in dispatch
    self.dispatch(h, type)
  File "/usr/local/lib64/python3.6/site-packages/proton/_events.py", line 164, in dispatch
    self.dispatch(h, type)
  File "/usr/local/lib64/python3.6/site-packages/proton/_events.py", line 161, in dispatch
    _dispatch(handler, type.method, self)
  File "/usr/local/lib64/python3.6/site-packages/proton/_events.py", line 128, in _dispatch
    m(*args)
  File "/usr/local/lib64/python3.6/site-packages/proton/_handlers.py", line 431, in on_connection_local_open
    self.on_connection_opened(event)
  File "/usr/local/lib64/python3.6/site-packages/proton/_handlers.py", line 471, in on_connection_opened
    _dispatch(self.delegate, 'on_connection_opened', event)
  File "/usr/local/lib64/python3.6/site-packages/proton/_events.py", line 128, in _dispatch
    m(*args)
  File "/home/runner/work/skupper-router/skupper-router/skupper-router/tests/system_tests_open_properties.py", line 54, in on_connection_opened
    self.open_properties = strip_default_options(event.connection.remote_properties)
  File "/home/runner/work/skupper-router/skupper-router/skupper-router/tests/system_tests_open_properties.py", line 37, in strip_default_options
    del options[key]
TypeError: 'NoneType' object does not support item deletion

https://github.com/skupperproject/skupper-router/runs/5397741867?check_suite_focus=true#step:27:423

Remove macOS code in CMakeLists and in the router

The unbearable optionality of dependencies and features

Qdrouterd build is very adaptable to the circumstances, switching off features and their tests depending on what's available. This is all well for a generic library, but may not be so great for an end-user application where uniformity can simplify some things greatly.

For example, should libwebsockets remain to be optional dependency? (I think not.) What about libnghttp2 (There I am not so sure, but I wouldn't leave it optional either.)
What about SKIP_DELETE_HTTP_LISTENER config option? (I'd decide if that should be ON or OFF and then made it that way; even if it requires messing with libwebsocket version, because the CentOS one does not suit, then so be it.)

What about Python version? RHEL8/CentOS Stream 8 ship with versions of Python ranging between 3.6 to 3.9. Is there a reason to use the oldest 3.6 version, which means some tests for TLS (that require 3.7) don't run?

Python test dependencies can be collected in a requirements-dev.txt file and always installed. That way tests wouldn't be skipping themselves on their own.

Port DISPATCH-2312

https://issues.apache.org/jira/browse/DISPATCH-2312

thread leak from qdrouterd on shutdown, reported from system_tests_one_router

https://github.com/skupperproject/skupper-router/runs/5406854344?check_suite_focus=true#step:27:807

18: E           WARNING: ThreadSanitizer: thread leak (pid=1676)
18: E             Thread T3 (tid=1687, finished) created by main thread at:
18: E               #0 pthread_create <null> (libtsan.so.0+0x5bef5)
18: E               #1 sys_thread /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:181 (qdrouterd+0x47a7bc)
18: E               #2 qd_server_run /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1489 (qdrouterd+0x4c2eac)
18: E               #3 main_process /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:105 (qdrouterd+0x423ccc)
18: E               #4 main /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:359 (qdrouterd+0x42318c)
18: E           
18: E             And 1 more similar thread leaks.
18: E           
18: E           SUMMARY: ThreadSanitizer: thread leak (/lib64/libtsan.so.0+0x5bef5) in pthread_create
18: E           ==================
18: E           ThreadSanitizer: reported 1 warnings

I don't rightly see how that possibly might happen

    for (i = 0; i < n; i++) {
        threads[i] = sys_thread(thread_run, qd_server);
    }
    thread_run(qd_server);      /* Use the current thread */
    for (i = 0; i < n; i++) {
        sys_thread_join(threads[i]);
        sys_thread_free(threads[i]);
    }
    free(threads);

Occasional failures in Http1AdaptorManagementTest (flakey test)

See https://github.com/apache/qpid-dispatch/runs/5218027151?check_suite_focus=true#step:27:133

Reduce sleep times in system_tests_http2

Do this so the test executes faster

Problem: package cyrus-sasl-md5-2.1.27-5.el8.x86_64 requires cyrus-sasl-lib(x86-64) = 2.1.27-5.el8, but none of the providers can be installed

https://github.com/skupperproject/skupper-router/runs/5333715965?check_suite_focus=true#step:3:143

Error: 
 Problem: package cyrus-sasl-md5-2.1.27-5.el8.x86_64 requires cyrus-sasl-lib(x86-64) = 2.1.27-5.el8, but none of the providers can be installed
  - cannot install both cyrus-sasl-lib-2.1.27-6.el8_5.x86_64 and cyrus-sasl-lib-2.1.27-5.el8.x86_64
  - package cyrus-sasl-devel-2.1.27-6.el8_5.x86_64 requires cyrus-sasl-lib(x86-64) = 2.1.27-6.el8_5, but none of the providers can be installed
  - cannot install the best candidate for the job
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
The command '/bin/sh -c dnf -y --setopt=tsflags=nodocs install gcc gcc-c++ make cmake cyrus-sasl-devel cyrus-sasl-plain cyrus-sasl-gssapi cyrus-sasl-md5 openssl-devel libuuid-devel swig wget patch findutils git valgrind libwebsockets-devel python3-devel libnghttp2-devel && dnf clean all -y' returned a non-zero code: 1