skupperproject / skupper-router Goto Github PK
View Code? Open in Web Editor NEWAn application-layer router for Skupper networks
Home Page: https://skupper.io
License: Apache License 2.0
An application-layer router for Skupper networks
Home Page: https://skupper.io
License: Apache License 2.0
There are many deprecated attributes and entities in the skupper-router schema. Remove these deprecated items along with any tests that test these.
Remove the link-routing support. This will greatly simplify testing and future development of new routing features. Most of the unique and valuable benefits of link-routing can be achieved using message streaming and non-terminal delivery dispositions.
There should not be any large gotchas, with maybe the exception of Policy CMP0065. CMake used to compile all binaries (under GCC) with -rdynamic
by default, but according to this new policy it removed the flag. Dispatch needs the flag so it can dlopen()
its own binary. Any tests that do router initializations also need the flag.
The solution is to either set ENABLE_EXPORTS
for each target that needs it, or to set CMAKE_ENABLE_EXPORTS
to get the old behavior.
According to discussion at #44 (comment), the -devel version can likely be replaced with nondevel version, which will make the resulting image slightly smaller.
Not reliably reproducible.
Build Info:
Os: Fedora 34
Skupper-router: commit b2a1748
Skupper-build: Debug (no runtime checks)
Proton: commit ec535e772e02bd67a34a90d301c450758f53833e
Proton build: RelWithDebInfo (no runtime checks)
$ ctest -V -j8
....
58: ======================================================================
58: ERROR: tearDownClass (system_tests_http2.Http2TestInteriorEdgeRouter)
58: ----------------------------------------------------------------------
58: Traceback (most recent call last):
58: File "/home/kgiusti/work/skupper/skupper-router/tests/system_test.py", line 851, in tearDownClass
58: cls.tester.teardown()
58: File "/home/kgiusti/work/skupper/skupper-router/tests/system_test.py", line 794, in teardown
58: raise RuntimeError("Errors during teardown: \n\n%s" % "\n\n".join([str(e) for e in errors]))
58: RuntimeError: Errors during teardown:
58:
58: Process 380029 error: exit code -11, expected 0
58: qdrouterd -c edge-router.conf -I /home/kgiusti/work/skupper/skupper-router/python
58: /home/kgiusti/work/skupper/skupper-router/BUILD/tests/system_test.dir/system_tests_http2/Http2TestInteriorEdgeRouter/setUpClass/edge-router-5\
2.cmd
$ gdb
...
Command Line: qdrouterd -c edge-router.conf -I /home/kgiusti/work/skupper/skupper-router/python
...
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000000000047b105 in qdr_delivery_message (delivery=0x9999999999999999)
at /home/kgiusti/work/skupper/skupper-router/src/router_core/delivery.c:176
176 return delivery->msg;
[Current thread is 1 (Thread 0x7f2e7302c640 (LWP 380032))]
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.33-21.fc34.x86_64 libdb-5.3.28-49.fc34.x86_64 libffi-3.1-28.fc34.x86_64 libgcc-11\
.2.1-9.fc34.x86_64 libselinux-3.2-1.fc34.x86_64 libxcrypt-4.4.27-2.fc34.x86_64 pcre2-10.36-4.fc34.x86_64 python3.10-3.10.2-1.fc34.x86_64
(gdb) bt
#0 0x000000000047b105 in qdr_delivery_message (delivery=0x9999999999999999)
at /home/kgiusti/work/skupper/skupper-router/src/router_core/delivery.c:176
#1 0x00000000004ae9a0 in read_data_callback (session=0x12da650, stream_id=19,
buf=0x12dc46a "B\003PUT\004\206b\237F\252S\377\206\306\305\304\017\r\002\062\060\300", length=16384, data_flags=0x7f2e7302ac40,
source=0x13ed5d0, user_data=0x126a910) at /home/kgiusti/work/skupper/skupper-router/src/adaptors/http2/http2_adaptor.c:1178
#2 0x00007f2e821df38b in nghttp2_session_pack_data (stream=0x17e3730, aux_data=0x13ed5d0, frame=0x13ed570, datamax=16384, bufs=0x12da7a8,
session=0x12da650) at /usr/src/debug/nghttp2-1.43.0-2.fc34.x86_64/lib/nghttp2_session.c:7173
#3 session_prep_frame (item=0x13ed570, session=0x12da650) at /usr/src/debug/nghttp2-1.43.0-2.fc34.x86_64/lib/nghttp2_session.c:2047
#4 nghttp2_session_mem_send_internal (session=session@entry=0x12da650, data_ptr=data_ptr@entry=0x7f2e7302ac90, fast_cb=fast_cb@entry=0)
at /usr/src/debug/nghttp2-1.43.0-2.fc34.x86_64/lib/nghttp2_session.c:2932
#5 0x00007f2e821e06a9 in nghttp2_session_send (session=0x12da650) at /usr/src/debug/nghttp2-1.43.0-2.fc34.x86_64/lib/nghttp2_session.c:3254
#6 0x00000000004b04b8 in qdr_http_delivery_update (context=0x11f8b40, dlv=0x15cf790, disp=38, settled=true)
at /home/kgiusti/work/skupper/skupper-router/src/adaptors/http2/http2_adaptor.c:1601
#7 0x0000000000471633 in qdr_connection_process (conn=0x127f250)
at /home/kgiusti/work/skupper/skupper-router/src/router_core/connections.c:390
#8 0x00000000004b4ac4 in handle_connection_event (e=0x11b2fe0, qd_server=0x1103fe0, context=0x126a910)
at /home/kgiusti/work/skupper/skupper-router/src/adaptors/http2/http2_adaptor.c:2545
#9 0x00000000004a294c in handle_event_with_context (e=0x11b2fe0, qd_server=0x1103fe0, context=0x126a910)
at /home/kgiusti/work/skupper/skupper-router/src/server.c:777
#10 0x00000000004a2989 in do_handle_raw_connection_event (e=0x11b2fe0, qd_server=0x1103fe0)
at /home/kgiusti/work/skupper/skupper-router/src/server.c:783
#11 0x00000000004a3813 in handle (qd_server=0x1103fe0, e=0x11b2fe0, pn_conn=0x0, ctx=0x0)
at /home/kgiusti/work/skupper/skupper-router/src/server.c:1060
#12 0x00000000004a3921 in thread_run (arg=0x1103fe0) at /home/kgiusti/work/skupper/skupper-router/src/server.c:1092
#13 0x00000000004660d3 in _thread_init (arg=0x119f790) at /home/kgiusti/work/skupper/skupper-router/src/posix/threading.c:172
#14 0x00007f2e825ef2a5 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f2e82104323 in clone () from /lib64/libc.so.6
...
(gdb) list
1173 nghttp2_data_source *source,
1174 void *user_data)
1175 {
1176 qdr_http2_connection_t *conn = (qdr_http2_connection_t *)user_data;
1177 qdr_http2_stream_data_t *stream_data = (qdr_http2_stream_data_t *)source->ptr;
1178 --> qd_message_t *message = qdr_delivery_message(stream_data->out_dlv);
1179 qd_message_depth_status_t status = qd_message_check_depth(message, QD_DEPTH_BODY);
1180
1181 // This flag tells nghttp2 that the data is not being copied into the buffer supplied by nghttp2 (uint8_t *buf).
1182 *data_flags |= NGHTTP2_DATA_FLAG_NO_COPY;
(gdb) p *conn
$3 = {context = {context = 0x126a910, handler = 0x4b44b6 <handle_connection_event>}, qdr_conn = 0x127f250, pn_raw_conn = 0x12bf690,
read_buffers = {{context = 0, bytes = 0x0, capacity = 0, size = 0, offset = 0}, {context = 0, bytes = 0x0, capacity = 0, size = 0,
offset = 0}, {context = 0, bytes = 0x0, capacity = 0, size = 0, offset = 0}, {context = 0, bytes = 0x0, capacity = 0, size = 0,
offset = 0}}, activate_timer = 0x11b47d0, ping_timer = 0x0, config = 0x125f610, server = 0x1103fe0, conn_id = 1, remote_address = 0x0,
stream_dispatcher = 0x1292bd0, stream_dispatcher_stream_data = 0x12a7ed0, stream_dispatcher_id = 1, data_prd = {source = {fd = 22196560,
ptr = 0x152b150}, read_callback = 0x4ae95e <read_data_callback>}, granted_read_buffs = {head = 0x131ee90, tail = 0x13686d0,
scratch = 0x0, size = 16}, prev_ping = 0, last_pn_raw_conn_read = 0, buffs = {head = 0x0, tail = 0x0, scratch = 0x13500d0, size = 0},
session = 0x12da650, streams = {head = 0x152b390, tail = 0x152b390, scratch = 0x152b390, size = 1}, connection_established = true,
grant_initial_buffers = false, ingress = false, timer_scheduled = false, client_magic_sent = true, woken_by_ping = false,
first_pinged = false, delete_egress_connections = false, goaway_received = false, raw_closed_read = 0, raw_closed_write = 0,
q2_blocked = false, q2_restart = 0, delay_buffer_write = 0, prev = 0x0, next = 0x0}
(gdb) p source
$4 = (nghttp2_data_source *) 0x13ed5d0
(gdb) p *source
$5 = {fd = 22196560, ptr = 0x152b150}
(gdb) p stream_data
$6 = (qdr_http2_stream_data_t *) 0x152b150
(gdb) p *stream_data
$7 = {conn = 0x9999999999999999, context = 0x9999999999999999,
reply_to = 0x9999999999999999 <error: Cannot access memory at address 0x9999999999999999>,
remote_site = 0x9999999999999999 <error: Cannot access memory at address 0x9999999999999999>,
method = 0x9999999999999999 <error: Cannot access memory at address 0x9999999999999999>,
request_status = 0x9999999999999999 <error: Cannot access memory at address 0x9999999999999999>, in_dlv = 0x9999999999999999,
out_dlv = 0x9999999999999999, incoming_id = 11068046444225730969, outgoing_id = 11068046444225730969,
out_dlv_local_disposition = 11068046444225730969, in_link = 0x9999999999999999, out_link = 0x9999999999999999, message = 0x9999999999999999,
app_properties = 0x9999999999999999, footer_properties = 0x9999999999999999, body_buffers = {head = 0x9999999999999999,
tail = 0x9999999999999999, scratch = 0x9999999999999999, size = 11068046444225730969}, curr_stream_data = 0x9999999999999999,
curr_stream_data_iter = 0x9999999999999999, next_stream_data = 0x9999999999999999, footer_stream_data = 0x9999999999999999,
footer_stream_data_iter = 0x9999999999999999, prev = 0x9999999999999999, next = 0x9999999999999999, curr_stream_data_result = 2576980377,
next_stream_data_result = 2576980377, payload_handled = -1717986919, in_link_credit = -1717986919, stream_id = -1717986919,
status = (QD_STREAM_HALF_CLOSED | unknown: 0x99999998), entire_footer_arrived = 153, entire_header_arrived = 153, out_msg_header_sent = 153,
out_msg_body_sent = 153, use_footer_properties = 153, full_payload_handled = 153, out_msg_has_body = 153, out_msg_data_flag_eof = 153,
out_msg_has_footer = 153, out_msg_send_complete = 153, disp_updated = 153, disp_applied = 153, header_and_props_composed = 153,
stream_force_closed = 153, in_dlv_decrefed = 153, out_dlv_decrefed = 153, body_data_added_to_msg = 153, bytes_in = -1717986919,
bytes_out = -1717986919, start = -7378697629483820647, stop = -7378697629483820647}
(gdb) quit
$ tail ./tests/system_test.dir/system_tests_http2/Http2TestInteriorEdgeRouter/setUpClass/edge-router.log
2022-02-17 10:35:21.924215 -0500 ROUTER_CORE (debug) [C2][L15][D77] Delivery outcome : is released (0x26) (/home/kgiusti/work/skupper/skupper-rou\
ter/src/router_core/delivery.c:392)
2022-02-17 10:35:21.924295 -0500 HTTP_ADAPTOR (trace) [C1] Activation triggered, calling pn_raw_connection_wake() (/home/kgiusti/work/skupper/sku\
pper-router/src/adaptors/http2/http2_adaptor.c:1705)
2022-02-17 10:35:21.924361 -0500 ROUTER_CORE (trace) Core action 'link_flow' (/home/kgiusti/work/skupper/skupper-router/src/router_core/router_co\
re_thread.c:235)
2022-02-17 10:35:21.924383 -0500 HTTP_ADAPTOR (trace) [C1] PN_RAW_CONNECTION_WAKE Wake-up (/home/kgiusti/work/skupper/skupper-router/src/adaptors\
/http2/http2_adaptor.c:2538)
2022-02-17 10:35:21.924410 -0500 ROUTER_CORE (trace) Core action 'link_detach' (/home/kgiusti/work/skupper/skupper-router/src/router_core/router_\
core_thread.c:235)
2022-02-17 10:35:21.924444 -0500 ROUTER_CORE (info) [C4][L7] Link lost: del=1 presett=0 psdrop=0 acc=1 rej=0 rel=0 mod=0 delay1=0 delay10=0 block\
ed=no (/home/kgiusti/work/skupper/skupper-router/src/router_core/connections.c:1090)
2022-02-17 10:35:21.924461 -0500 ROUTER_CORE (trace) Core action 'link_detach' (/home/kgiusti/work/skupper/skupper-router/src/router_core/router_\
core_thread.c:235)
2022-02-17 10:35:21.924479 -0500 ROUTER_CORE (info) [C4][L8] Link lost: del=1 presett=1 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 block\
ed=no (/home/kgiusti/work/skupper/skupper-router/src/router_core/connections.c:1090)
2022-02-17 10:35:21.924499 -0500 ROUTER_CORE (trace) Core action 'connection_closed' (/home/kgiusti/work/skupper/skupper-router/src/router_core/r\
outer_core_thread.c:235)
2022-02-17 10:35:21.924515 -0500 ROUTER_CORE (info) [C4] Connection Closed (/home/kgiusti/work/skupper/skupper-router/src/router_core/connections\
.c:1562)
[kgiusti@localhost BUILD (ISSUE-79)]$
Best addressed after resolving #13 first.
I've very occasionally seen a crash on the following assert, when running the scenario from https://issues.apache.org/jira/browse/DISPATCH-2192. I would like to investigate further and maybe come up with reliable reproduction steps. As far as I can tell, I was doing, in some order, with some repetitions
iperf3 -c 127.0.0.1 -p 8000
iperf3 -c 127.0.0.1 -p 8000 -P 100 -n 9999999999999
, ^C
^C
'd a router, then started it againqdrouterd: /home/jdanek/repos/skupper-router/src/message.c:1066: void qd_message_free(qd_message_t*): Assertion `DEQ_IS_EMPTY(msg->stream_data_list)' failed.
There was the same problem in qpid-dispatch and there it also was hard to reliably reproduce, https://issues.apache.org/jira/browse/DISPATCH-2165.
In order to support end-to-end flow control and other more advanced routing features, the MAU (Mobile Address Update) message format needs to be enhanced such that data can be carried along with addresses. Formerly, the existence or non-existence of an address was the extent of the data encoded in this message.
This is a breaking (non backward compatible) change to the protocol and should be completed prior to the first official release.
self = <system_tests_one_router.OneRouterTest testMethod=test_30_presettled_overflow>
def test_30_presettled_overflow(self):
test = PresettledOverflowTest(self.address)
test.run()
> self.assertIsNone(test.error)
AssertionError: 'No deliveries were dropped' is not None
https://github.com/skupperproject/skupper-router/runs/5186887345?check_suite_focus=true#step:27:117
Use the default system version of python instead.
Need to remove this step from the workflow configuration.
POC: determine code and performance impact.
Things I'll need to get merged here
And that's all
/home/runner/work/skupper-router/skupper-router/skupper-router/tests/system_tests_http2.py:168: AssertionError
____ Http2TestEdgeToEdgeViaInteriorRouter.test_post_upload_large_image_jpg _____
self = <system_tests_http2.Http2TestEdgeToEdgeViaInteriorRouter testMethod=test_post_upload_large_image_jpg>
@unittest.skipIf(skip_test(), "Python 3.7 or greater, Quart 0.13.0 or greater and curl needed to run http2 tests")
@unittest.skipIf(skip_test_in_ci('QPID_SYSTEM_TEST_SKIP_HTTP2_LARGE_IMAGE_UPLOAD_TEST'), skip_reason)
def test_post_upload_large_image_jpg(self):
# curl -X POST -H "Content-Type: multipart/form-data" -F "data=@/home/gmurthy/opensource/test.jpg"
# http://127.0.0.1:9000/upload --http2-prior-knowledge
address = self.router_qdra.http_addresses[0] + "/upload"
out = self.run_curl(address, args=['-X', 'POST', '-H', 'Content-Type: multipart/form-data',
'-F', 'data=@' + image_file('test.jpg')])
> self.assertIn('Success', out)
E AssertionError: 'Success' not found in '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>408 Request Timeout</title>\n<h1>Request Timeout</h1>\n<p>The server closed the network connection because the browser didn't finish the request within the specified time.</p>\n'
https://github.com/skupperproject/skupper-router/runs/5350854560?check_suite_focus=true#step:27:4354
Observed in current skupper-router image
/home/qdrouterd/bin/configure.sh: line 56: python3: command not found
DISPATCH-2132: atomic access to core uptime counter
44: =================================== FAILURES ===================================
44: __________ OneRouterLinkCountersTest.test_06_large_message_released ___________
44: 44: self = <system_tests_delivery_counts.OneRouterLinkCountersTest testMethod=test_06_large_message_released>
44:
44: def test_06_large_message_released(self):
44: > self.verify_released(True)
44: 44: /home/runner/work/skupper-router/skupper-router/skupper-router/tests/system_tests_delivery_counts.py:1198:
44: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
44: /home/runner/work/skupper-router/skupper-router/skupper-router/tests/system_tests_delivery_counts.py:1059: in verify_released
44: self.assertEqual(test.receiver_stats['releasedCount'], self.COUNT)
44: E AssertionError: 29 != 40
44: - generated xml file: /__w/skupper-router/skupper-router/skupper-router/build/tests/junitxmls/system_tests_delivery_counts.xml -
44: =========================== short test summary info ============================
44: FAILED ::OneRouterLinkCountersTest::test_06_large_message_released - Assertio...
44: ======================== 1 failed, 26 passed in 59.43s =========================
23/33 Test #44: system_tests_delivery_counts ...................***Failed 59.76 sec
============================= test session starts ==============================
https://github.com/ganeshmurthy/skupper-router/runs/5109959880?check_suite_focus=true#step:27:1023
osx
Currently test_q2_block_unblock does the following
self.assertGreater(num_blocked, 0)
self.assertGreater(num_unblocked, 0)
It should also do the following -
self.assertGreaterEqual(num_unblocked, num_blocked)
This is a follow up to (my) comments on #44. This ticket suggests to consider doing (and eventually do) one or more of the following
-fstack-protector
and with -D_FORTIFY_SOURCE=2
-DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON
and -fno-semantic-interposition
(https://fedoraproject.org/wiki/Changes/PythonNoSemanticInterpositionSpeedup)
-lSegFault
in production build to always dump stacktrace on crash (use addr2line
to later decrypt the trace)LIBC_FATAL_STDERR_=1
LWS_WITH_LIBUV=ON
and LWS_WITH_HTTP2=OFF
?
set -Eeuo pipefail
QDROUTERD_AUTO_MESH_DISCOVERY
as core skupper-router feature, instead of hacky config file editing during image startup18: E WARNING: ThreadSanitizer: data race (pid=1997)
18: E Read of size 1 at 0x7b44000277d5 by thread T1:
18: E #0 qdr_delivery_continue_CT /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/delivery.c:1108 (qdrouterd+0x4945e2)
18: E #1 router_core_thread /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/router_core_thread.c:236 (qdrouterd+0x4a2e5a)
18: E #2 _thread_init /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:172 (qdrouterd+0x47a2fd)
18: E
18: E Previous write of size 1 at 0x7b44000277d5 by thread T3:
18: E #0 qdr_delivery_set_presettled /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/delivery.c:136 (qdrouterd+0x4924d5)
18: E #1 AMQP_rx_handler /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_node.c:526 (qdrouterd+0x4baa28)
18: E #2 deferred_AMQP_rx_handler /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_node.c:882 (qdrouterd+0x4bb142)
18: E #3 invoke_deferred_calls /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:764 (qdrouterd+0x4bd1db)
18: E #4 invoke_deferred_calls /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:753 (qdrouterd+0x4c0466)
18: E #5 handle /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1017 (qdrouterd+0x4c0466)
18: E #6 thread_run /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1092 (qdrouterd+0x4c21c4)
18: E #7 _thread_init /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:172 (qdrouterd+0x47a2fd)
18: E
18: E Location is heap block of size 320 at 0x7b4400027740 allocated by thread T4:
18: E #0 posix_memalign <null> (libtsan.so.0+0x32a23)
18: E #1 qd_alloc /home/runner/work/skupper-router/skupper-router/skupper-router/src/alloc_pool.c:391 (qdrouterd+0x447b69)
18: E #2 new_qdr_delivery_t /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/delivery.c:24 (qdrouterd+0x491f4c)
18: E #3 qdr_link_deliver /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/transfer.c:63 (qdrouterd+0x4a8305)
18: E #4 AMQP_rx_handler /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_node.c:837 (qdrouterd+0x4bab23)
18: E #5 AMQP_link_detach_handler /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_node.c:1089 (qdrouterd+0x4bb0ba)
18: E #6 AMQP_link_detach_handler /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_node.c:1062 (qdrouterd+0x4bb0ba)
18: E #7 close_links /home/runner/work/skupper-router/skupper-router/skupper-router/src/container.c:328 (qdrouterd+0x458349)
18: E #8 qd_container_handle_event /home/runner/work/skupper-router/skupper-router/skupper-router/src/container.c:520 (qdrouterd+0x45b234)
18: E #9 handle /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1067 (qdrouterd+0x4c048e)
18: E #10 thread_run /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1092 (qdrouterd+0x4c2144)
18: E #11 _thread_init /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:172 (qdrouterd+0x47a2fd)
18: E
18: E Thread T1 (tid=2004, running) created by main thread at:
18: E #0 pthread_create <null> (libtsan.so.0+0x5bef5)
18: E #1 sys_thread /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:181 (qdrouterd+0x47a78c)
18: E #2 qdr_core /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/router_core.c:121 (qdrouterd+0x49d920)
18: E #3 qd_router_setup_late /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_node.c:2078 (qdrouterd+0x4bb8a8)
18: E #4 ffi_call_unix64 <null> (libffi.so.6+0x6c03)
18: E #5 main_process /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:92 (qdrouterd+0x423c3c)
18: E #6 main /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:359 (qdrouterd+0x42315c)
18: E
18: E Thread T3 (tid=2007, running) created by main thread at:
18: E #0 pthread_create <null> (libtsan.so.0+0x5bef5)
18: E #1 sys_thread /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:181 (qdrouterd+0x47a78c)
18: E #2 qd_server_run /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1484 (qdrouterd+0x4c2d7c)
18: E #3 main_process /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:105 (qdrouterd+0x423c9c)
18: E #4 main /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:359 (qdrouterd+0x42315c)
18: E
18: E Thread T4 (tid=2009, running) created by main thread at:
18: E #0 pthread_create <null> (libtsan.so.0+0x5bef5)
18: E #1 sys_thread /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:181 (qdrouterd+0x47a78c)
18: E #2 qd_server_run /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1484 (qdrouterd+0x4c2d7c)
18: E #3 main_process /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:105 (qdrouterd+0x423c9c)
18: E #4 main /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:359 (qdrouterd+0x42315c)
18: E
18: E SUMMARY: ThreadSanitizer: data race /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/delivery.c:1108 in qdr_delivery_continue_CT
18: E ==================
18: E ThreadSanitizer: reported 1 warnings
https://github.com/skupperproject/skupper-router/runs/5186886991?check_suite_focus=true#step:27:894
A separate logging/tracing facility is needed for the protocol adaptors to serve users that are more accustomed to network/firewall/router logs and are not concerned with AMQP/messaging/link-protocol details.
This feature will make the use of the protocol adaptors more observable and debuggable to network-familiar developers and operators.
https://github.com/jiridanek/skupper-router/runs/5026451378?check_suite_focus=true#step:27:2618
72: E AssertionError: 'Stuck delivery: At least one delivery' unexpectedly found in '2022-02-01 19:03:49.081570 +0000 ROUTER_CORE (info) [C2][L10] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 3 seconds (/home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/modules/stuck_delivery_detection/delivery_tracker.c:60)\n' : Stuck deliveries should not be logged!
72:
72: /home/runner/work/skupper-router/skupper-router/skupper-router/tests/system_tests_tcp_adaptor.py:1119: AssertionError
72: - generated xml file: /__w/skupper-router/skupper-router/skupper-router/build/tests/junitxmls/system_tests_tcp_adaptor.xml -
72: =========================== short test summary info ============================
72: FAILED ::TcpAdaptorStuckDeliveryTest::test_01_ignore_stuck_deliveries - Asser...
72: =================== 1 failed, 9 passed, 2 skipped in 49.22s ====================
35/37 Test #72: system_tests_tcp_adaptor ..........................***Failed 49.43 sec
This is something that does not appear to be previously logged in Dispatch Apache Jira.
Configure the router as in system_tests_http2.py::Http2TestOneStandaloneRouter::test_post_upload_large_image_jpg
, then repeatedly run
curl --http2-prior-knowledge http://127.0.0.1:21581/upload -X POST -H 'Content-Type: multipart/form-data' -F data=@/home/jdanek/repos/skupper-router/tests/images/test.jpg
=================================================================
==909120==ERROR: AddressSanitizer: use-after-poison on address 0x614000047ef0 at pc 0x000000530e58 bp 0x7f360fb20900 sp 0x7f360fb208f8
READ of size 8 at 0x614000047ef0 thread T3
#0 0x530e57 in qd_message_extend /home/jdanek/repos/skupper-router/src/message.c:2399
#1 0x53c318 in qd_message_stream_data_append /home/jdanek/repos/skupper-router/src/message.c:3028
#2 0x68ab09 in on_data_chunk_recv_callback /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:560
#3 0x7f3625a6c753 in nghttp2_session_mem_recv (/lib64/libnghttp2.so.14+0xe753)
#4 0x6946cb in handle_incoming_http /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2133
#5 0x6b1852 in handle_connection_event /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2542
#6 0x662fd4 in handle_event_with_context /home/jdanek/repos/skupper-router/src/server.c:780
#7 0x662fd4 in do_handle_raw_connection_event /home/jdanek/repos/skupper-router/src/server.c:786
#8 0x662fd4 in handle /home/jdanek/repos/skupper-router/src/server.c:1063
#9 0x66a6a5 in thread_run /home/jdanek/repos/skupper-router/src/server.c:1095
#10 0x7f3624f6fb19 in start_thread (/lib64/libc.so.6+0x8db19)
#11 0x7f3624ff464f in __GI___clone3 (/lib64/libc.so.6+0x11264f)
0x614000047ef0 is located 176 bytes inside of 384-byte region [0x614000047e40,0x614000047fc0)
allocated by thread T4 here:
#0 0x7f3625f654fc in __interceptor_posix_memalign (/lib64/libasan.so.6+0xaf4fc)
#1 0x4bb49d in qd_alloc /home/jdanek/repos/skupper-router/src/alloc_pool.c:391
#2 0x522268 in new_qd_message_t /home/jdanek/repos/skupper-router/src/message.c:93
#3 0x522268 in qd_message /home/jdanek/repos/skupper-router/src/message.c:1011
#4 0x68329c in create_http2_stream_data /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:478
#5 0x6ae964 in qdr_http_deliver /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2032
#6 0x611d26 in qdr_link_process_deliveries /home/jdanek/repos/skupper-router/src/router_core/transfer.c:194
#7 0x58a621 in qdr_connection_process /home/jdanek/repos/skupper-router/src/router_core/connections.c:401
#8 0x6affbe in handle_connection_event /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2545
#9 0x662fd4 in handle_event_with_context /home/jdanek/repos/skupper-router/src/server.c:780
#10 0x662fd4 in do_handle_raw_connection_event /home/jdanek/repos/skupper-router/src/server.c:786
#11 0x662fd4 in handle /home/jdanek/repos/skupper-router/src/server.c:1063
#12 0x66a6a5 in thread_run /home/jdanek/repos/skupper-router/src/server.c:1095
#13 0x7f3624f6fb19 in start_thread (/lib64/libc.so.6+0x8db19)
Thread T3 created by T0 here:
#0 0x7f3625f0c866 in pthread_create (/lib64/libasan.so.6+0x56866)
#1 0x55f155 in sys_thread /home/jdanek/repos/skupper-router/src/posix/threading.c:181
#2 0x66cc1c in qd_server_run /home/jdanek/repos/skupper-router/src/server.c:1489
#3 0x6b633d in main_process /home/jdanek/repos/skupper-router/router/src/main.c:105
#4 0x432332 in main /home/jdanek/repos/skupper-router/router/src/main.c:359
#5 0x7f3624f0f55f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f)
Thread T4 created by T0 here:
#0 0x7f3625f0c866 in pthread_create (/lib64/libasan.so.6+0x56866)
#1 0x55f155 in sys_thread /home/jdanek/repos/skupper-router/src/posix/threading.c:181
#2 0x66cc1c in qd_server_run /home/jdanek/repos/skupper-router/src/server.c:1489
#3 0x6b633d in main_process /home/jdanek/repos/skupper-router/router/src/main.c:105
#4 0x432332 in main /home/jdanek/repos/skupper-router/router/src/main.c:359
#5 0x7f3624f0f55f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f)
SUMMARY: AddressSanitizer: use-after-poison /home/jdanek/repos/skupper-router/src/message.c:2399 in qd_message_extend
Shadow bytes around the buggy address:
0x0c2880000f80: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
0x0c2880000f90: 00 00 00 00 00 00 00 00 00 00 f7 f7 f7 f7 f7 f7
0x0c2880000fa0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
0x0c2880000fb0: f7 f7 f7 f7 00 00 00 00 fa fa fa fa fa fa fa fa
0x0c2880000fc0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
=>0x0c2880000fd0: 00 00 00 00 00 00 00 00 00 00 f7 f7 f7 f7[f7]f7
0x0c2880000fe0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
0x0c2880000ff0: f7 f7 f7 f7 00 00 00 00 fa fa fa fa fa fa fa fa
0x0c2880001000: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c2880001010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c2880001020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==909120==ABORTING
From the same test, I've also seen the following. See last leak for addr2line
decoded trace
addr2line --exe router/qdrouterd --pretty-print 0x<addr>
alloc.c: Items of type 'qd_http2_buffer_t' remain allocated at shutdown: 10
Leak: 2022-02-27 14:54:31.996089 +0100 type: qd_http2_buffer_t address: 0x62900007d290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]
Leak: 2022-02-27 14:54:31.996285 +0100 type: qd_http2_buffer_t address: 0x629000082290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]
Leak: 2022-02-27 14:54:31.996481 +0100 type: qd_http2_buffer_t address: 0x629000168290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]
Leak: 2022-02-27 14:54:31.996670 +0100 type: qd_http2_buffer_t address: 0x6290000af290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]
Leak: 2022-02-27 14:54:31.996820 +0100 type: qd_http2_buffer_t address: 0x6290000e1290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]
Leak: 2022-02-27 14:54:31.996987 +0100 type: qd_http2_buffer_t address: 0x6290002e9290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]
Leak: 2022-02-27 14:54:31.997152 +0100 type: qd_http2_buffer_t address: 0x629000325290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]
Leak: 2022-02-27 14:54:31.997321 +0100 type: qd_http2_buffer_t address: 0x629000069290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]
Leak: 2022-02-27 14:54:31.997487 +0100 type: qd_http2_buffer_t address: 0x6290001ef290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]
Leak: 2022-02-27 14:54:31.997619 +0100 type: qd_http2_buffer_t address: 0x6290001e0290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce] /home/jdanek/repos/skupper-router/src/alloc_pool.c:349
qdrouterd() [0x697522] /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:51
qdrouterd() [0x698111] /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:154
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca] /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2173
qdrouterd() [0x6b036e] /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2555
qdrouterd() [0x662fd5] /home/jdanek/repos/skupper-router/src/server.c:780
qdrouterd() [0x66a6a6] /home/jdanek/repos/skupper-router/src/server.c:1095
::OpenPropertiesConfigTest::test_02_verify_connector_properties FAILED
::OpenPropertiesQdManageTest::test_01_verify_listener_properties PASSED
::OpenPropertiesQdManageTest::test_02_verify_connector_properties PASSED
::OpenPropertiesBadConfigTest::test_01_invalid_properties_check PASSED
::OpenPropertiesBadConfigTest::test_02_invalid_role_check PASSEDException in thread Thread-7:
Traceback (most recent call last):
File "/usr/lib64/python3.6/threading.py", line 937, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 885, in run
self._target(*self._args, **self._kwargs)
File "/home/runner/work/skupper-router/skupper-router/skupper-router/tests/test_broker.py", line 113, in _main
while self._container.process():
File "/usr/local/lib64/python3.6/site-packages/proton/_reactor.py", line 257, in process
event.dispatch(handler)
File "/usr/local/lib64/python3.6/site-packages/proton/_events.py", line 164, in dispatch
self.dispatch(h, type)
File "/usr/local/lib64/python3.6/site-packages/proton/_events.py", line 164, in dispatch
self.dispatch(h, type)
File "/usr/local/lib64/python3.6/site-packages/proton/_events.py", line 161, in dispatch
_dispatch(handler, type.method, self)
File "/usr/local/lib64/python3.6/site-packages/proton/_events.py", line 128, in _dispatch
m(*args)
File "/usr/local/lib64/python3.6/site-packages/proton/_handlers.py", line 431, in on_connection_local_open
self.on_connection_opened(event)
File "/usr/local/lib64/python3.6/site-packages/proton/_handlers.py", line 471, in on_connection_opened
_dispatch(self.delegate, 'on_connection_opened', event)
File "/usr/local/lib64/python3.6/site-packages/proton/_events.py", line 128, in _dispatch
m(*args)
File "/home/runner/work/skupper-router/skupper-router/skupper-router/tests/system_tests_open_properties.py", line 54, in on_connection_opened
self.open_properties = strip_default_options(event.connection.remote_properties)
File "/home/runner/work/skupper-router/skupper-router/skupper-router/tests/system_tests_open_properties.py", line 37, in strip_default_options
del options[key]
TypeError: 'NoneType' object does not support item deletion
https://github.com/skupperproject/skupper-router/runs/5397741867?check_suite_focus=true#step:27:423
Qdrouterd build is very adaptable to the circumstances, switching off features and their tests depending on what's available. This is all well for a generic library, but may not be so great for an end-user application where uniformity can simplify some things greatly.
SKIP_DELETE_HTTP_LISTENER
config option? (I'd decide if that should be ON or OFF and then made it that way; even if it requires messing with libwebsocket version, because the CentOS one does not suit, then so be it.)What about Python version? RHEL8/CentOS Stream 8 ship with versions of Python ranging between 3.6 to 3.9. Is there a reason to use the oldest 3.6 version, which means some tests for TLS (that require 3.7) don't run?
requirements-dev.txt
file and always installed. That way tests wouldn't be skipping themselves on their own.https://github.com/skupperproject/skupper-router/runs/5406854344?check_suite_focus=true#step:27:807
18: E WARNING: ThreadSanitizer: thread leak (pid=1676)
18: E Thread T3 (tid=1687, finished) created by main thread at:
18: E #0 pthread_create <null> (libtsan.so.0+0x5bef5)
18: E #1 sys_thread /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:181 (qdrouterd+0x47a7bc)
18: E #2 qd_server_run /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:1489 (qdrouterd+0x4c2eac)
18: E #3 main_process /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:105 (qdrouterd+0x423ccc)
18: E #4 main /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:359 (qdrouterd+0x42318c)
18: E
18: E And 1 more similar thread leaks.
18: E
18: E SUMMARY: ThreadSanitizer: thread leak (/lib64/libtsan.so.0+0x5bef5) in pthread_create
18: E ==================
18: E ThreadSanitizer: reported 1 warnings
I don't rightly see how that possibly might happen
for (i = 0; i < n; i++) {
threads[i] = sys_thread(thread_run, qd_server);
}
thread_run(qd_server); /* Use the current thread */
for (i = 0; i < n; i++) {
sys_thread_join(threads[i]);
sys_thread_free(threads[i]);
}
free(threads);
Do this so the test executes faster
https://github.com/skupperproject/skupper-router/runs/5333715965?check_suite_focus=true#step:3:143
Error:
Problem: package cyrus-sasl-md5-2.1.27-5.el8.x86_64 requires cyrus-sasl-lib(x86-64) = 2.1.27-5.el8, but none of the providers can be installed
- cannot install both cyrus-sasl-lib-2.1.27-6.el8_5.x86_64 and cyrus-sasl-lib-2.1.27-5.el8.x86_64
- package cyrus-sasl-devel-2.1.27-6.el8_5.x86_64 requires cyrus-sasl-lib(x86-64) = 2.1.27-6.el8_5, but none of the providers can be installed
- cannot install the best candidate for the job
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
The command '/bin/sh -c dnf -y --setopt=tsflags=nodocs install gcc gcc-c++ make cmake cyrus-sasl-devel cyrus-sasl-plain cyrus-sasl-gssapi cyrus-sasl-md5 openssl-devel libuuid-devel swig wget patch findutils git valgrind libwebsockets-devel python3-devel libnghttp2-devel && dnf clean all -y' returned a non-zero code: 1
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.