microsoft / msquic Goto Github PK

Cross-platform, C implementation of the IETF QUIC protocol, exposed to C, C++, C# and Rust.

License: MIT License

C 61.70% C++ 21.87% Roff 1.95% CMake 1.77% PowerShell 5.40% Dockerfile 0.11% Shell 0.15% C# 5.93% Batchfile 0.38% Rust 0.74%

quic network-programming protocol c secure cross-platform cpp csharp rust

msquic's Introduction

MsQuic is a Microsoft implementation of the IETF QUIC protocol. It is cross-platform, written in C and designed to be a general purpose QUIC library. MsQuic also has C++ API wrapper classes and exposes interop layers for both Rust and C#.

Protocol Features

QUIC has many benefits when compared to existing "TLS over TCP" scenarios:

All packets are encrypted and handshake is authenticated with TLS 1.3.
Parallel streams of (reliable and unreliable) application data.
Exchange application data in the first round trip (0-RTT).
Improved congestion control and loss recovery.
Survives a change in the clients IP address or port.
Stateless load balancing.
Easily extendable for new features and extensions.

Library Features

MsQuic has several features that differentiates it from other QUIC implementations:

Optimized for client and server.
Optimized for maximal throughput and minimal latency.
Asynchronous IO.
Receive side scaling (RSS) support.
UDP send and receive coalescing support.

Documentation

For platform support details, see the Platforms docs.
For release details, see the Release docs.
For performance data, see the Performance dashboard.
For building the library, see the Build docs.
For testing the library, see the Testing docs.
For using the API, see the API docs or the Sample.
For running a sample server and client app, see the Quick Start Guide.
For deploying QUIC, see the Deployment docs.
For diagnosing issues, see the Diagnostics docs and the Trouble Shooting Guide.
For other frequently asked questions, see the FAQs.

Contributing

For information on contributing, please see our contribution guidelines. Feel free to take a look at our Good First Issues list if you're looking for somewhere to start. If you'd just like to talk, come chat with us on Discord.

msquic's People

Contributors

Stargazers

Watchers

Forkers

5gapp am11 zzroot dio leejw51 xanathar shyamalschandra hide5stm pzx601917159 xuliang mbrukman peerix ethan-arrowood azureexpert linecode wujs wlm328cs dut3062796s doytsujin yang123vc baiyfcu newlysoft feiser2016 zjm-tech gc19861210 thurday zhangyangisme b-xiang danieldin95 thinkbyrj blond2005 8ggmaker azurementor rheehot fuzhound marshalyuan yanyongshan dlyshare qq0827 anxornot leolovemary shzhouqijian 18901233704 ldjking xvmoon fengpsky cquptchongqing packet-high-performance helloliupan feiyunwill jiangbiaodeng stanleyw-tw eds709394 simondong1979 samtsai2 ptpt52 516025 xiaming9880 brinkqiang2cpp jingsao binaryworld fungchou supython-coder lxq2537664558 zhe1234zou leiminchn tomzhang asdlei99 studytnot feiwei9696 mohammatmajken cser2016 woodenmen oesh ivanzz1001 lancyxd wellertian xfeng986 mycopy hyungjic ccnjust bofa1ex dreamwind1985 forestmo thadhouse larseggert ahsan27 anna-vardanyan ma-dada andrisetyo163-yahoo-com crazyfanstr youngniki ricardoforever rita0405 szijani emmanuel-p1 yuliyayevtukh ledinhtrong123 406345 rocker9527

msquic's Issues

Binary Logging Format for Linux

Currently, Linux uses syslog, which is extremely slow, because all events result in sprintf calls to create strings out of the data and then passed to syslog to write to disk. The current idea is to add support for LTTng.

Build Windows Kernel Mode (Local)

Now that the code is all external we need to get it all building.

Add runtime platform support detection

It would be useful to have consistent error code(s) for MsQuicOpen to return when the current platform doesn't have a minimum Windows/OpenSSL version, registry for TLS 1.3 isn't enabled, etc.

This would enable users to prompt the end-user, log, etc. with an obvious message indicating what they might do to get support.

Support OSX

Along with Windows and Linux, we need OSX support as well.

Enable Code Analysis in Builds

Code Analysis should be enabled for the core and platform code in the Windows kernel VS project files. To enable this by default, all existing warnings will need to be fixed so the build will pass.

To do this, the project files simply need the following to be added to them:

  <PropertyGroup>
    <RunCodeAnalysis>true</RunCodeAnalysis>
  </PropertyGroup>

Event Tests Fail Occasionally

The Event tests that validate certain events are received in a particular order fail occasionally. After debugging this, I found that the ideal processor changed event was confusing the tests. We need to just ignore these events when testing.

Build and Run Unittests

We have code for platform and core unit tests that were originally written for TAEF. They need to be moved over to gtest, made cross platform, added to the build and on boarded for automated testing.

StreamShutdown Should be Callable on all Streams

Right now, StreamShutdown fails on a stream that's not yet started or not completely started. This creates a burned on the API caller to synchronize state to wait for start to complete (if necessary). It's trivial for msquic to do the synchronization instead, as all the pieces are already in place for it.

Windows Build Needs to Generate PDB for Release EXEs

To be able to ingest a binary, the Windows build needs both the binary and PDB file. Currently, msquictest.exe cannot be ingested because the PDB for release builds isn't generated.

TLS State Prematurely Discarded on Server

A bug was discovered during HTTP/3 interop where (on server) TLS state was being thrown away before indicating the CONNECTED event to the app. The following logic is currently used to decide when to throw away the TLS state:

Is server
Is connected
All crypto data acknowledged

The code was currently depending on (3) above to prevent the premature discard of TLS state, because a NewSessionTicket message is usually sent after the handshake completes, so there would still be unacknowledged crypto data outstanding when we indicate the CONNECTED event to the app. This breaks down if no NST is sent.

Windows User Mode ARM/AMR64 Builds Fail

Currently, it looks like the primary problem is linking to ETW and Registry functions. Hopefully as simple as pointing to the correct dependent libraries.

PS G:\msquic> .\build.ps1 -Arch arm -Config Release
[02/11/2020 08:14:05] Generating files...
[02/11/2020 08:14:05] cmake -g 'Visual Studio 16 2019' -A arm -DQUIC_ARCH=arm -DQUIC_TLS=schannel ../../..
-- Source Dir: G:/msquic
-- Host System name: Windows
-- The C compiler identification is MSVC 19.24.28315.0
-- The CXX compiler identification is MSVC 19.24.28315.0
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.24.28314/bin/Hostx64/arm/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.24.28314/bin/Hostx64/arm/cl.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.24.28314/bin/Hostx64/arm/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.24.28314/bin/Hostx64/arm/cl.exe -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- System name: Windows
-- System version: 10.0.18362.0
-- Platform version: 10.0.18362.0
-- Configuring for manifested ETW events and logging
-- Disabling 0-RTT support
-- Configuring for SChannel
-- Found PythonInterp: C:/Program Files (x86)/Python36-32/python.exe (found version "3.6.5")
-- Looking for pthread.h
-- Looking for pthread.h - not found
-- Found Threads: TRUE
-- Configuring done
-- Generating done
-- Build files have been written to: G:/msquic/bld/windows/arm_schannel
[02/11/2020 08:14:22] Building...
[02/11/2020 08:14:22] cmake --build . --config Release
Microsoft (R) Build Engine version 16.4.0+e901037fe for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

  Checking Build System
  Generating inc/MsQuicEtw.h, inc/MsQuicEtw.rc
  Building Custom Rule G:/msquic/CMakeLists.txt
  Building Custom Rule G:/msquic/src/core/CMakeLists.txt
  ack_tracker.c
  api.c
  binding.c
  congestion_control.c
  connection.c
  crypto.c
  crypto_tls.c
  frame.c
  library.c
  listener.c
  lookup.c
  loss_detection.c
  operation.c
  packet.c
  packet_builder.c
  packet_space.c
  path.c
  range.c
  recv_buffer.c
  registration.c
  send.c
  send_buffer.c
  sent_packet_metadata.c
  session.c
  settings.c
  stream.c
  stream_recv.c
  stream_send.c
  stream_set.c
  timer_wheel.c
  worker.c
  core.vcxproj -> G:\msquic\bld\windows\arm_schannel\obj\Release\core.lib
  Building Custom Rule G:/msquic/submodules/googletest/googletest/CMakeLists.txt
  gtest-all.cc
  gtest.vcxproj -> G:\msquic\bld\windows\arm_schannel\lib\Release\gtest.lib
  Building Custom Rule G:/msquic/submodules/googletest/googletest/CMakeLists.txt
  gtest_main.cc
  gtest_main.vcxproj -> G:\msquic\bld\windows\arm_schannel\lib\Release\gtest_main.lib
  Building Custom Rule G:/msquic/src/platform/CMakeLists.txt
  datapath_winuser.c
  hashtable.c
  platform_winuser.c
  storage_winuser.c
  cert_capi.c
  selfsign_capi.c
  tls_schannel.c
  platform.vcxproj -> G:\msquic\bld\windows\arm_schannel\obj\Release\platform.lib
  Building Custom Rule G:/msquic/src/bin/CMakeLists.txt
  dllmain.c
LINK : warning LNK4075: ignoring '/INCREMENTAL' due to '/OPT:ICF' specification [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
     Creating library G:/msquic/bld/windows/arm_schannel/obj/Release/msquic.lib and object G:/msquic/bld/windows/arm_schannel/obj/Release/msquic.exp
platform.lib(hashtable.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
platform.lib(cert_capi.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
platform.lib(platform_winuser.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
platform.lib(datapath_winuser.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
platform.lib(storage_winuser.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
platform.lib(tls_schannel.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(loss_detection.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(crypto.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(packet_builder.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(crypto_tls.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(frame.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(ack_tracker.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(packet_space.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(congestion_control.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(stream_recv.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(stream_set.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(path.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(send_buffer.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(range.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(recv_buffer.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(send.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(stream_send.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(stream.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(connection.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(packet.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(timer_wheel.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(session.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(listener.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(api.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(worker.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(library.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(settings.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(binding.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
core.lib(registration.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
platform.lib(platform_winuser.obj) : error LNK2019: unresolved external symbol __imp_EventRegister referenced in function McGenEventRegister [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
platform.lib(platform_winuser.obj) : error LNK2019: unresolved external symbol __imp_EventUnregister referenced in function McGenEventUnregister [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
platform.lib(storage_winuser.obj) : error LNK2019: unresolved external symbol __imp_RegCloseKey referenced in function QuicStorageClose [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
platform.lib(storage_winuser.obj) : error LNK2019: unresolved external symbol __imp_RegNotifyChangeKeyValue referenced in function QuicStorageOpen [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
platform.lib(storage_winuser.obj) : error LNK2019: unresolved external symbol __imp_RegOpenKeyExA referenced in function QuicStorageOpen [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
platform.lib(storage_winuser.obj) : error LNK2019: unresolved external symbol __imp_RegQueryValueExA referenced in function QuicStorageReadValue [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
G:\msquic\artifacts\windows\arm_Release_schannel\msquic.dll : fatal error LNK1120: 7 unresolved externals [G:\msquic\bld\windows\arm_schannel\src\bin\msquic.vcxproj]
  Building Custom Rule G:/msquic/src/tools/etwlib/CMakeLists.txt
  QuicEventCollection.cpp
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\include\vector(726,1): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc [G:\msquic\bld\windows\arm_sc
hannel\src\tools\etwlib\msquicetw.vcxproj]
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\include\vector(701): message : see reference to function template instantiation 'QuicApiData *std::vector<QuicApiData,std::allocator<QuicApiData>>:
:_Emplace_reallocate<_Ty>(QuicApiData *const ,_Ty &&)' being compiled [G:\msquic\bld\windows\arm_schannel\src\tools\etwlib\msquicetw.vcxproj]
          with
          [
              _Ty=QuicApiData
          ]
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\include\vector(709): message : see reference to function template instantiation 'QuicApiData *std::vector<QuicApiData,std::allocator<QuicApiData>>:
:_Emplace_reallocate<_Ty>(QuicApiData *const ,_Ty &&)' being compiled [G:\msquic\bld\windows\arm_schannel\src\tools\etwlib\msquicetw.vcxproj]
          with
          [
              _Ty=QuicApiData
          ]
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\include\vector(722): message : see reference to function template instantiation '_Ty &std::vector<_Ty,std::allocator<_Ty>>::emplace_back<QuicApiDat
a>(QuicApiData &&)' being compiled [G:\msquic\bld\windows\arm_schannel\src\tools\etwlib\msquicetw.vcxproj]
          with
          [
              _Ty=QuicApiData
          ]
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\include\vector(721): message : while compiling class template member function 'void std::vector<QuicApiData,std::allocator<QuicApiData>>::push_back
(_Ty &&)' [G:\msquic\bld\windows\arm_schannel\src\tools\etwlib\msquicetw.vcxproj]
          with
          [
              _Ty=QuicApiData
          ]
G:\msquic\src\tools\etwlib\QuicEventCollection.cpp(469): message : see reference to function template instantiation 'void std::vector<QuicApiData,std::allocator<QuicApiData>>::push_back(_Ty &&)' being compiled [G:\msquic\bld\windows\arm
_schannel\src\tools\etwlib\msquicetw.vcxproj]
          with
          [
              _Ty=QuicApiData
          ]
G:\msquic\src\tools\etwlib\QuicEventCollection.cpp(469): message : see reference to class template instantiation 'std::vector<QuicApiData,std::allocator<QuicApiData>>' being compiled [G:\msquic\bld\windows\arm_schannel\src\tools\etwlib\
msquicetw.vcxproj]
  msquicetw.vcxproj -> G:\msquic\bld\windows\arm_schannel\obj\Release\msquicetw.lib
  Building Custom Rule G:/msquic/src/test/lib/CMakeLists.txt
  ApiTest.cpp
  DrillDescriptor.cpp
  EventTest.cpp
  QuicDrill.cpp
  QuicTest.cpp
  TestConnection.cpp
  TestListener.cpp
  TestStream.cpp
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\include\vector(726,1): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc (compiling source file G:\msq
uic\src\test\lib\QuicDrill.cpp) [G:\msquic\bld\windows\arm_schannel\src\test\lib\testlib.vcxproj]
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\include\vector(701): message : see reference to function template instantiation 'unsigned char *std::vector<uint8_t,std::allocator<uint8_t>>::_Empl
ace_reallocate<_Ty>(unsigned char *const ,_Ty &&)' being compiled [G:\msquic\bld\windows\arm_schannel\src\test\lib\testlib.vcxproj]
          with
          [
              _Ty=unsigned char
          ] (compiling source file G:\msquic\src\test\lib\QuicDrill.cpp)
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\include\vector(709): message : see reference to function template instantiation 'unsigned char *std::vector<uint8_t,std::allocator<uint8_t>>::_Empl
ace_reallocate<_Ty>(unsigned char *const ,_Ty &&)' being compiled [G:\msquic\bld\windows\arm_schannel\src\test\lib\testlib.vcxproj]
          with
          [
              _Ty=unsigned char
          ] (compiling source file G:\msquic\src\test\lib\QuicDrill.cpp)
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\include\vector(722): message : see reference to function template instantiation '_Ty &std::vector<_Ty,std::allocator<_Ty>>::emplace_back<unsigned c
har>(unsigned char &&)' being compiled [G:\msquic\bld\windows\arm_schannel\src\test\lib\testlib.vcxproj]
          with
          [
              _Ty=uint8_t
          ] (compiling source file G:\msquic\src\test\lib\QuicDrill.cpp)
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\include\vector(721): message : while compiling class template member function 'void std::vector<uint8_t,std::allocator<uint8_t>>::push_back(_Ty &&)
' [G:\msquic\bld\windows\arm_schannel\src\test\lib\testlib.vcxproj]
          with
          [
              _Ty=uint8_t
          ] (compiling source file G:\msquic\src\test\lib\QuicDrill.cpp)
G:\msquic\src\test\lib\QuicDrill.cpp(410): message : see reference to function template instantiation 'void std::vector<uint8_t,std::allocator<uint8_t>>::push_back(_Ty &&)' being compiled [G:\msquic\bld\windows\arm_schannel\src\test\lib
\testlib.vcxproj]
          with
          [
              _Ty=uint8_t
          ]
G:\msquic\src\test\lib\DrillDescriptor.h(77): message : see reference to class template instantiation 'std::vector<uint8_t,std::allocator<uint8_t>>' being compiled (compiling source file G:\msquic\src\test\lib\QuicDrill.cpp) [G:\msquic\
bld\windows\arm_schannel\src\test\lib\testlib.vcxproj]
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\include\vector(726,1): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc (compiling source file G:\msq
uic\src\test\lib\DrillDescriptor.cpp) [G:\msquic\bld\windows\arm_schannel\src\test\lib\testlib.vcxproj]
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\include\vector(701): message : see reference to function template instantiation 'unsigned char *std::vector<uint8_t,std::allocator<uint8_t>>::_Empl
ace_reallocate<_Ty>(unsigned char *const ,_Ty &&)' being compiled [G:\msquic\bld\windows\arm_schannel\src\test\lib\testlib.vcxproj]
          with
          [
              _Ty=unsigned char
          ] (compiling source file G:\msquic\src\test\lib\DrillDescriptor.cpp)
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\include\vector(709): message : see reference to function template instantiation 'unsigned char *std::vector<uint8_t,std::allocator<uint8_t>>::_Empl
ace_reallocate<_Ty>(unsigned char *const ,_Ty &&)' being compiled [G:\msquic\bld\windows\arm_schannel\src\test\lib\testlib.vcxproj]
          with
          [
              _Ty=unsigned char
          ] (compiling source file G:\msquic\src\test\lib\DrillDescriptor.cpp)
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\include\vector(722): message : see reference to function template instantiation '_Ty &std::vector<_Ty,std::allocator<_Ty>>::emplace_back<unsigned c
har>(unsigned char &&)' being compiled [G:\msquic\bld\windows\arm_schannel\src\test\lib\testlib.vcxproj]
          with
          [
              _Ty=uint8_t
          ] (compiling source file G:\msquic\src\test\lib\DrillDescriptor.cpp)
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\include\vector(721): message : while compiling class template member function 'void std::vector<uint8_t,std::allocator<uint8_t>>::push_back(_Ty &&)
' [G:\msquic\bld\windows\arm_schannel\src\test\lib\testlib.vcxproj]
          with
          [
              _Ty=uint8_t
          ] (compiling source file G:\msquic\src\test\lib\DrillDescriptor.cpp)
G:\msquic\src\test\lib\DrillDescriptor.cpp(21): message : see reference to function template instantiation 'void std::vector<uint8_t,std::allocator<uint8_t>>::push_back(_Ty &&)' being compiled [G:\msquic\bld\windows\arm_schannel\src\tes
t\lib\testlib.vcxproj]
          with
          [
              _Ty=uint8_t
          ]
G:\msquic\src\test\lib\DrillDescriptor.h(77): message : see reference to class template instantiation 'std::vector<uint8_t,std::allocator<uint8_t>>' being compiled (compiling source file G:\msquic\src\test\lib\DrillDescriptor.cpp) [G:\m
squic\bld\windows\arm_schannel\src\test\lib\testlib.vcxproj]
  testlib.vcxproj -> G:\msquic\bld\windows\arm_schannel\obj\Release\testlib.lib
  Building Custom Rule G:/msquic/src/tools/attack/CMakeLists.txt
  attack.cpp
  packet_writer.cpp
platform.lib(storage_winuser.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
platform.lib(hashtable.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(loss_detection.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(crypto.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(packet_builder.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(crypto_tls.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(send_buffer.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(ack_tracker.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(packet_space.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(congestion_control.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(stream_send.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(stream_recv.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(stream_set.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(path.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(connection.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(timer_wheel.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(recv_buffer.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(send.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(listener.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(api.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(worker.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(stream.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(settings.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(binding.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(registration.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(session.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
platform.lib(datapath_winuser.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
platform.lib(tls_schannel.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
platform.lib(cert_capi.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(library.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(frame.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(range.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
core.lib(packet.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
platform.lib(platform_winuser.obj) : error LNK2001: unresolved external symbol __imp_EventWriteTransfer [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
platform.lib(platform_winuser.obj) : error LNK2019: unresolved external symbol __imp_EventRegister referenced in function McGenEventRegister [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
platform.lib(platform_winuser.obj) : error LNK2019: unresolved external symbol __imp_EventUnregister referenced in function McGenEventUnregister [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
platform.lib(storage_winuser.obj) : error LNK2019: unresolved external symbol __imp_RegCloseKey referenced in function QuicStorageClose [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
platform.lib(storage_winuser.obj) : error LNK2019: unresolved external symbol __imp_RegNotifyChangeKeyValue referenced in function QuicStorageOpen [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
platform.lib(storage_winuser.obj) : error LNK2019: unresolved external symbol __imp_RegOpenKeyExA referenced in function QuicStorageOpen [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
platform.lib(storage_winuser.obj) : error LNK2019: unresolved external symbol __imp_RegQueryValueExA referenced in function QuicStorageReadValue [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]
G:\msquic\artifacts\windows\arm_Release_schannel\quicattack.exe : fatal error LNK1120: 7 unresolved externals [G:\msquic\bld\windows\arm_schannel\src\tools\attack\quicattack.vcxproj]

DrillInitialPacketCids & DrillInitialPacketToken Tests Failing on Linux

These tests seem to have been failing on Linux since they were added.

Error Message:

/home/vsts/work/1/s/test/lib/QuicDrill.cpp:312
DroppedPacketsAfter - DroppedPacketsBefore not equal to 1

Server Must Always Process Retry Tokens

Currently, the server code (QuicBindingShouldRetryConnection) only processes a retry token if it's at the memory threshold. But this is incorrect. If the client sends a token, it's going to validate the server sends the original_connection_id TP in response. So the server needs to extract that from the token regardless.

This will mostly consist of shuffling around the logic a bit. First check if there's a token. If there is, process it. Only if there isn't check if we need one.

0-RTT for Interop Server Fails

Describe the bug

During draft-27 interop testing, we discovered that 0-RTT doesn't seem to be succeeding against our public server (client side works just fine against other servers). Both our own testing via quicinterop.exe and other clients are failing.

Steps to reproduce the behavior

Run quicinterop.exe -target:msquic
View output
Expect to see the Z included in the passing test cases.

Expected vs actual behavior

0-RTT packets should get acknowledged during the test, but they always end up needing to be retransmitted.

Related to this, we need to double check that 0-RTT tests were ported over from TAEF correctly, and are running.

Support Client Certificates

Client scenarios:

Single cert: The user specifies one certificate before creating a new connection. That cert is sent (if the server asks for it?).
Multiple certs: The user provides a list of possible certificates. The implementation selects one from the list that matches the issuers list provided by the server. (Compare to SslStream's client cert list).
- Once the connection is established the user can call an API to find out which cert was used.
Dynamic cert: The caller provides a callback that is invoked when the server asks for a certificate. The input is the issuers list (is there any other data available?) and the result is an optional certificate. (Compare to SslStream's callback).

If MSQuic provided only the dynamic option then the caller could implement the other two on top.

Server scenarios:

Settings can be provided per connection after seeing the SNI, ALPN, and other Client Hello data.
Client certs can be set to Required, Requested/Optional, or Disabled/Ignored.
Flags for what types of validation to enable/disable for the client cert.
A callback that is shown the validation results and can accept or reject the cert and/or connection. Compare to SslStream's callback.
Once the connection is established the server user can call an API to retrieve the client cert that was negotiated, if any.
~~Dynamic renegotiation to retrieve a client cert~~ - Not supported by TLS 1.3.

Unassigned Server Connections Leaked on Shutdown

While running spinquic, I hit the timer wheel assert (here) indicating there were still connections left in a worker's timer wheel on shutdown. The problem is that there was an idle (no queued operations) connection currently assigned to one of the global/library workers. When the library gets uninitialized it cleans up the library workers. The logic here attempts to clean up any leftover connections, but this logic only loops over active (queued operations) connections; and missing idle connections.

I think the fix will be that somehow we need to track the currently idle connections on these workers and makes sure to explicitly clean them up. This is only needed for idle connections in the library workers, because these connections have no external (i.e. app) references yet, so it's the library's responsibility to clean them up.

Add Option for Ciphers to QUIC_CREDENTIAL_CONFIG

Compare to CipherSuitesPolicy.

The client and server need the ability in code to restrict which ciphers can be negotiated. Additionally, the server needs to be able to make this decision per connection, after receiving the Client Hello. E.g. HTTP/2 required negotiating from a restricted set of ciphers, but you wouldn't know if the connection was using HTTP/2 until the ALPN information was received.

Historically ciphers have only been configurable at the OS level and that's still true on Windows/Schannel. OpenSsl however provides the ability to specify a list of allowed ciphers in code that overrides the global settings.

This capability is useful when phasing in new ciphers or phasing out old ones. Doing so machine wide risks destabilizing too many applications at once, services want to update apps one at a time.

Tests Failures in Callbacks Result in Assertion Failures

Generally, when a test validation fails in a callback, it seems to take ~500 ms. This is more than the currently configured 200 ms "error" timeout after which we trigger an assert. This then causes the entire test to crash right there and a dump is collecting. I have been leaning towards completely removing the asserts, but for the time being we should at least just increase the timeout.

Doc bug in build.md

Doc bug:
Additional Requirements on Windows
Visual Studio
2019 or higher Latest Windows Insider builds.

Add: Cmake

Document Platform Support Level

We need to make sure the current support level for different platforms and TLS libraries is accurately documented.

Tests Crash if Initialization Fails

The test setup code wasn't properly terminating the test where setup fails.

Assertion Failure in msquic!QuicConnStart

SpinQuic occasionally hits an assert that seems to indicate that somehow a client connection is getting started more than once. I'm not sure how this could be possible. The dump and symbols can be found here:

https://microsoft.visualstudio.com/OS/_build/results?buildId=21336292&view=artifacts&type=publishedArtifacts

Assertion D:\a\1\s\src\core\connection.c(1439): Path->Binding == ((void *)0)

0:004> kn
 # Child-SP          RetAddr           Call Site
00 000000c7`dfdff690 00007ff9`8655e76e msquic!QuicConnStart+0xe8 [D:\a\1\s\src\core\connection.c @ 1439] 
01 000000c7`dfdff740 00007ff9`865504f1 msquic!QuicConnProcessApiOperation+0xae [D:\a\1\s\src\core\connection.c @ 5034] 
02 000000c7`dfdff790 00007ff9`8654bdc8 msquic!QuicConnDrainOperations+0x321 [D:\a\1\s\src\core\connection.c @ 5191] 
03 000000c7`dfdff930 00007ff9`8654ab67 msquic!QuicWorkerProcessConnection+0x248 [D:\a\1\s\src\core\worker.c @ 432] 
04 000000c7`dfdffaf0 00007ff9`c2637974 msquic!QuicWorkerThread+0xd7 [D:\a\1\s\src\core\worker.c @ 513] 
05 000000c7`dfdffc90 00007ff9`c2b0a271 kernel32!BaseThreadInitThunk+0x14 [base\win32\client\thread.c @ 64] 
06 000000c7`dfdffcc0 00000000`00000000 ntdll!RtlUserThreadStart+0x21 [minkernel\ntdll\rtlstrt.c @ 1163] 

0:004> dt /r Connection
Local var @ 0xc7dfdff740 Type QUIC_CONNECTION*
0x0000029e`de00fbb0 
   +0x000 Type             : 3 ( QUIC_HANDLE_TYPE_CLIENT )
   +0x008 ClientContext    : 0x0000029e`dd846860 Void
   +0x010 SessionLink      : QUIC_LIST_ENTRY
      +0x000 Flink            : 0x0000029e`dda62a68 QUIC_LIST_ENTRY
      +0x008 Blink            : 0x0000029e`de32edd0 QUIC_LIST_ENTRY
   +0x020 WorkerLink       : QUIC_LIST_ENTRY
      +0x000 Flink            : 0x0000029e`dda60d70 QUIC_LIST_ENTRY
      +0x008 Blink            : 0x0000029e`dda60d70 QUIC_LIST_ENTRY
   +0x030 TimerLink        : QUIC_LIST_ENTRY
      +0x000 Flink            : (null) 
      +0x008 Blink            : (null) 
   +0x040 Worker           : 0x0000029e`dda60d00 QUIC_WORKER
      +0x000 Enabled          : 0x1 ''
      +0x001 IsActive         : 0x1 ''
      +0x002 IdealProcessor   : 0 ''
      +0x004 ThreadID         : 0x10a4
      +0x008 AverageQueueDelay : 0
      +0x010 TimerWheel       : QUIC_TIMER_WHEEL
         +0x000 NextExpirationTime : 0x1e2dda3a
         +0x008 ConnectionCount  : 0x4b
         +0x010 NextConnection   : 0x0000029e`de0053e0 QUIC_CONNECTION
         +0x018 SlotCount        : 0x20
         +0x020 Slots            : 0x0000029e`dda611b0 QUIC_LIST_ENTRY
      +0x038 Ready            : 0x00000000`00000128 Void
      +0x040 Thread           : 0x00000000`0000012c Void
      +0x048 Lock             : _RTL_CRITICAL_SECTION
      +0x070 Connections      : QUIC_LIST_ENTRY
         +0x000 Flink            : 0x0000029e`dda60d70 QUIC_LIST_ENTRY
         +0x008 Blink            : 0x0000029e`dda60d70 QUIC_LIST_ENTRY
      +0x080 Operations       : QUIC_LIST_ENTRY
         +0x000 Flink            : 0x0000029e`dda60d80 QUIC_LIST_ENTRY
         +0x008 Blink            : 0x0000029e`dda60d80 QUIC_LIST_ENTRY
      +0x090 OperationCount   : 0
      +0x098 DroppedOperationCount : 0
      +0x0a0 StreamPool       : QUIC_POOL
      +0x0c0 SendRequestPool  : QUIC_POOL
      +0x0e0 SentPacketPool   : QUIC_SENT_PACKET_POOL
      +0x1e0 ApiContextPool   : QUIC_POOL
      +0x200 StatelessContextPool : QUIC_POOL
      +0x220 OperPool         : QUIC_POOL
   +0x048 Registration     : 0x0000029e`dda60c00 QUIC_REGISTRATION
      +0x000 Type             : 0 ( QUIC_HANDLE_TYPE_REGISTRATION )
      +0x008 ClientContext    : (null) 
      +0x010 PartitionCount   : 0x2 ''
      +0x011 ExecProfileType  : 0 ''
      +0x012 CidPrefixLength  : 0 ''
      +0x018 CidPrefix        : (null) 
      +0x020 Link             : QUIC_LIST_ENTRY
         +0x000 Flink            : 0x00007ff9`865e3160 QUIC_LIST_ENTRY
         +0x008 Blink            : 0x00007ff9`865e3160 QUIC_LIST_ENTRY
      +0x030 WorkerPool       : 0x0000029e`dda60cc0 QUIC_WORKER_POOL
         +0x000 WorkerCount      : 0x2 ''
         +0x001 LastWorker       : 0 ''
         +0x040 Workers          : [0] QUIC_WORKER
      +0x038 Lock             : _RTL_CRITICAL_SECTION
         +0x000 DebugInfo        : 0xffffffff`ffffffff _RTL_CRITICAL_SECTION_DEBUG
         +0x008 LockCount        : 0n-1
         +0x00c RecursionCount   : 0n0
         +0x010 OwningThread     : (null) 
         +0x018 LockSemaphore    : (null) 
         +0x020 SpinCount        : 0x20007d0
      +0x060 Sessions         : QUIC_LIST_ENTRY
         +0x000 Flink            : 0x0000029e`dda61628 QUIC_LIST_ENTRY
         +0x008 Blink            : 0x0000029e`dda63368 QUIC_LIST_ENTRY
      +0x070 SecConfigRundown : QUIC_RUNDOWN_REF
         +0x000 RefCount         : 0n2
         +0x008 RundownComplete  : 0x00000000`00000124 Void
      +0x080 AppName          : [0]  "spinquic"
   +0x050 Session          : 0x0000029e`dda62990 QUIC_SESSION
      +0x000 Type             : 1 ( QUIC_HANDLE_TYPE_SESSION )
      +0x008 ClientContext    : (null) 
      +0x010 Registration     : 0x0000029e`dda60c00 QUIC_REGISTRATION
      +0x018 Link             : QUIC_LIST_ENTRY
         +0x000 Flink            : 0x0000029e`dda63368 QUIC_LIST_ENTRY
         +0x008 Blink            : 0x0000029e`dda61fe8 QUIC_LIST_ENTRY
      +0x028 Rundown          : QUIC_RUNDOWN_REF
         +0x000 RefCount         : 0n30
         +0x008 RundownComplete  : 0x00000000`00000144 Void
      +0x038 TlsSession       : 0x0000029e`dda63310 QUIC_TLS_SESSION
         +0x000 AlpnLength       : 5
         +0x008 Alpn             : [0] 0xababab32`6e697073  "--- memory read error at address 0xababab32`6e697073 ---"
      +0x040 CompartmentId    : 1
      +0x048 AppSpecificStorage : (null) 
      +0x050 Settings         : QUIC_SETTINGS
         +0x000 PacingDefault    : 0x1 ''
         +0x001 MaxPartitionCount : 0x40 '@'
         +0x002 MaxOperationsPerDrain : 0x10 ''
         +0x004 RetryMemoryLimit : 0x41
         +0x008 MaxWorkerQueueDelayUs : 0x3d090
         +0x00c MaxStatelessOperations : 0x10
         +0x010 InitialWindowPackets : 0xa
         +0x014 SendIdleTimeoutMs : 0x3e8
         +0x018 InitialRttMs     : 0x1f4
         +0x01c MaxAckDelayMs    : 0x19
         +0x020 DisconnectTimeoutMs : 0x3e80
         +0x024 KeepAliveIntervalMs : 0
         +0x028 HandshakeIdleTimeoutMs : 0x2710
         +0x030 IdleTimeoutMs    : 0x7530
         +0x038 BidiStreamCount  : 0
         +0x03a UnidiStreamCount : 5
         +0x03c TlsClientMaxSendBuffer : 0x2000
         +0x040 TlsServerMaxSendBuffer : 0
         +0x044 StreamRecvWindowDefault : 0x8000
         +0x048 StreamRecvBufferDefault : 0x1000
         +0x04c ConnFlowControlWindow : 0x1000000
         +0x050 MaxBytesPerKey   : 0x00000040`00000000
         +0x058 AppSet           : <unnamed-tag>
      +0x0b0 ServerCache      : QUIC_HASHTABLE
         +0x000 Flags            : 0
         +0x004 TableSize        : 0x80
         +0x008 NumEntries       : 1
         +0x00c NonEmptyBuckets  : 1
         +0x010 NumEnumerators   : 0
         +0x018 Directory        : 0x0000029e`dda62ae0 Void
         +0x018 SecondLevelDir   : 0x0000029e`dda62ae0 QUIC_LIST_ENTRY
         +0x018 FirstLevelDir    : 0x0000029e`dda62ae0  -> 0x0000029e`dda62ae0 QUIC_LIST_ENTRY
      +0x0d0 ServerCacheLock  : _RTL_SRWLOCK
         +0x000 Ptr              : (null) 
      +0x0d8 Connections      : QUIC_LIST_ENTRY
         +0x000 Flink            : 0x0000029e`de0b1e00 QUIC_LIST_ENTRY
         +0x008 Blink            : 0x0000029e`de00fbc0 QUIC_LIST_ENTRY
      +0x0e8 ConnectionsLock  : _RTL_CRITICAL_SECTION
      +0x110 AlpnLength       : 0x5 ''
      +0x111 Alpn             : [0]  "spin2"
   +0x058 RefCount         : 0n3
   +0x05c State            : QUIC_CONNECTION_STATE
      +0x000 Flags            : 0x1cc3817
      +0x000 Allocated        : 0y1
      +0x000 Initialized      : 0y1
      +0x000 Started          : 0y1
      +0x000 Connected        : 0y0
      +0x000 ClosedLocally    : 0y1
      +0x000 ClosedRemotely   : 0y0
      +0x000 AppClosed        : 0y0
      +0x000 HandleShutdown   : 0y0
      +0x001 HandleClosed     : 0y0
      +0x001 Uninitialized    : 0y0
      +0x001 Freed            : 0y0
      +0x001 EncryptionEnabled : 0y1
      +0x001 HeaderProtectionEnabled : 0y1
      +0x001 ExternalOwner    : 0y1
      +0x001 GotFirstServerResponse : 0y0
      +0x001 ReceivedRetryPacket : 0y0
      +0x002 HandshakeConfirmed : 0y0
      +0x002 ListenerAccepted : 0y0
      +0x002 LocalAddressSet  : 0y1
      +0x002 RemoteAddressSet : 0y1
      +0x002 UpdateWorker     : 0y0
      +0x002 ShutdownCompleteTimedOut : 0y0
      +0x002 SendShutdownCompleteNotif : 0y1
      +0x002 UseSendBuffer    : 0y1
      +0x003 UsePacing        : 0y1
      +0x003 ShareBinding     : 0y0
      +0x003 TestTransportParameterSet : 0y0
   +0x060 WorkerThreadID   : 0x10a4
   +0x064 ServerCertValidationFlags : 0
   +0x068 AllocProcIndex   : 0x1 ''
   +0x069 ServerID         : 0 ''
   +0x06a PartitionID      : 0xe0 ''
   +0x06b DestCidCount     : 0x1 ''
   +0x06c PathsCount       : 0x1 ''
   +0x06d NextPathId       : 0x1 ''
   +0x06e WorkerProcessing : 0y1
   +0x06e HasQueuedWork    : 0y0
   +0x06f OutFlowBlockedReasons : 0 ''
   +0x070 AckDelayExponent : 0x8 ''
   +0x074 MaxAckDelayMs    : 0x19
   +0x078 IdleTimeoutMs    : 0x7530
   +0x080 HandshakeIdleTimeoutMs : 0x2710
   +0x088 DisconnectTimeoutUs : 0xf42400
   +0x08c KeepAliveIntervalMs : 0
   +0x090 NextSourceCidSequenceNumber : 1
   +0x098 RetirePriorTo    : 0
   +0x0a0 Paths            : [4] QUIC_PATH
      +0x000 ID               : 0 ''
      +0x001 IsActive         : 0y1
      +0x001 InitiatedCidUpdate : 0y0
      +0x001 GotFirstRttSample : 0y0
      +0x001 GotValidPacket   : 0y0
      +0x001 IsPeerValidated  : 0y1
      +0x001 SpinBit          : 0y0
      +0x001 SendChallenge    : 0y0
      +0x001 SendResponse     : 0y0
      +0x002 Mtu              : 0x500
      +0x008 Binding          : 0x0000029e`dde3b4a0 QUIC_BINDING
      +0x010 LocalAddress     : _SOCKADDR_INET
         +0x000 Ipv4             : sockaddr_in
         +0x000 Ipv6             : sockaddr_in6
         +0x000 si_family        : 2
      +0x02c RemoteAddress    : _SOCKADDR_INET
         +0x000 Ipv4             : sockaddr_in
         +0x000 Ipv6             : sockaddr_in6
         +0x000 si_family        : 2
      +0x048 DestCid          : 0x0000029e`dd95e070 QUIC_CID_QUIC_LIST_ENTRY
         +0x000 Link             : QUIC_LIST_ENTRY
         +0x010 ResetToken       : [16]  ""
         +0x020 CID              : QUIC_CID
      +0x050 Allowance        : 0xffffffff
      +0x054 SmoothedRtt      : 0x7a120
      +0x058 MinRtt           : 0xffffffff
      +0x05c MaxRtt           : 0
      +0x060 RttVariance      : 0
      +0x064 LatestRttSample  : 0
      +0x068 Response         : [8]  ""
      +0x070 Challenge        : [8]  ""
   +0x280 SourceCids       : QUIC_SINGLE_LIST_ENTRY
      +0x000 Next             : 0x0000029e`dd924678 QUIC_SINGLE_LIST_ENTRY
         +0x000 Next             : (null) 
   +0x288 DestCids         : QUIC_LIST_ENTRY
      +0x000 Flink            : 0x0000029e`dd95e070 QUIC_LIST_ENTRY
      +0x008 Blink            : 0x0000029e`dd95e070 QUIC_LIST_ENTRY
   +0x298 OrigCID          : (null) 
   +0x2a0 Timers           : [6] QUIC_CONN_TIMER_ENTRY
      +0x000 Type             : 0 ( QUIC_CONN_TIMER_PACING )
      +0x008 ExpirationTime   : 0xffffffff`ffffffff
   +0x300 ReceiveQueueCount : 0
   +0x308 ReceiveQueue     : (null) 
   +0x310 ReceiveQueueTail : 0x0000029e`de00feb8  -> (null) 
   +0x318 ReceiveQueueLock : _RTL_CRITICAL_SECTION
   +0x340 OperQ            : QUIC_OPERATION_QUEUE
      +0x000 ActivelyProcessing : 0x1 ''
      +0x008 Lock             : _RTL_CRITICAL_SECTION
         +0x000 DebugInfo        : 0xffffffff`ffffffff _RTL_CRITICAL_SECTION_DEBUG
         +0x008 LockCount        : 0n-1
         +0x00c RecursionCount   : 0n0
         +0x010 OwningThread     : (null) 
         +0x018 LockSemaphore    : (null) 
         +0x020 SpinCount        : 0x20007d0
      +0x030 List             : QUIC_LIST_ENTRY
         +0x000 Flink            : 0x000000c7`e00ff328 QUIC_LIST_ENTRY
         +0x008 Blink            : 0x000000c7`e00ff328 QUIC_LIST_ENTRY
   +0x380 BackUpOper       : QUIC_OPERATION
   +0x3b8 BackupApiContext : QUIC_API_CONTEXT
   +0x3f0 BackUpOperUsed   : 0
   +0x3f4 CloseStatus      : 80410002
   +0x3f8 CloseErrorCode   : 0x116
   +0x400 CloseReasonPhrase : (null) 
   +0x408 RemoteServerName : 0x0000029e`dde0ab60  "127.0.0.1"
   +0x410 PeerTransportParams : QUIC_TRANSPORT_PARAMETERS
      +0x000 Flags            : 0x59ff
      +0x008 IdleTimeout      : 0x7530
      +0x010 InitialMaxStreamDataBidiLocal : 0x8000
      +0x018 InitialMaxStreamDataBidiRemote : 0x8000
      +0x020 InitialMaxStreamDataUni : 0x8000
      +0x028 InitialMaxData   : 0x1000000
      +0x030 InitialMaxBidiStreams : 2
      +0x038 InitialMaxUniStreams : 1
      +0x040 MaxPacketSize    : 0x5c0
      +0x048 AckDelayExponent : 8
      +0x050 MaxAckDelay      : 0x29
      +0x058 ActiveConnectionIdLimit : 4
      +0x060 StatelessResetToken : [16]  ""
      +0x070 PreferredAddress : _SOCKADDR_INET
         +0x000 Ipv4             : sockaddr_in
         +0x000 Ipv6             : sockaddr_in6
         +0x000 si_family        : 0
      +0x08c OriginalConnectionID : [20]  ""
      +0x0a0 OriginalConnectionIDLength : 0 ''
   +0x4b8 DecodedAckRanges : QUIC_RANGE
      +0x000 SubRanges        : 0x0000029e`dd8a8a30 QUIC_SUBRANGE
         +0x000 Low              : 0xbaadf00d`baadf00d
         +0x008 Count            : 0xbaadf00d`baadf00d
      +0x008 UsedLength       : 0
      +0x00c AllocLength      : 8
      +0x010 MaxAllocSize     : 0x1000
   +0x4d0 Streams          : QUIC_STREAM_SET
      +0x000 Types            : [4] QUIC_STREAM_TYPE_INFO
         +0x000 MaxTotalStreamCount : 2
         +0x008 TotalStreamCount : 0
         +0x010 MaxCurrentStreamCount : 0
         +0x012 CurrentStreamCount : 0
      +0x060 StreamTable      : (null) 
      +0x068 ClosedStreams    : QUIC_LIST_ENTRY
         +0x000 Flink            : 0x0000029e`de0100e8 QUIC_LIST_ENTRY
         +0x008 Blink            : 0x0000029e`de0100e8 QUIC_LIST_ENTRY
   +0x548 CongestionControl : QUIC_CONGESTION_CONTROL
      +0x000 HasHadCongestionEvent : 0y0
      +0x000 IsInRecovery     : 0y0
      +0x000 IsInPersistentCongestion : 0y0
      +0x000 TimeOfLastAckValid : 0y0
      +0x004 InitialWindowPackets : 0xa
      +0x008 SendIdleTimeoutMs : 0x3e8
      +0x00c CongestionWindow : 0x3200
      +0x010 SlowStartThreshold : 0xffffffff
      +0x014 BytesInFlight    : 0
      +0x018 BytesInFlightMax : 0x1900
      +0x01c Exemptions       : 0 ''
      +0x020 TimeOfLastAck    : 0
      +0x028 TimeOfCongAvoidStart : 0
      +0x030 KCubic           : 0
      +0x034 WindowMax        : 0
      +0x038 WindowLastMax    : 0
      +0x040 RecoverySentPacketNumber : 0
   +0x590 LossDetection    : QUIC_LOSS_DETECTION
      +0x000 PacketsInFlight  : 0
      +0x008 LargestAck       : 0
      +0x010 LargestAckEncryptLevel : 0 ( QUIC_ENCRYPT_LEVEL_INITIAL )
      +0x018 LargestSentPacketNumber : 0
      +0x020 SentPackets      : (null) 
      +0x028 SentPacketsTail  : 0x0000029e`de010160  -> (null) 
      +0x030 TimeOfLastPacketSent : 0
      +0x038 LostPackets      : (null) 
      +0x040 LostPacketsTail  : 0x0000029e`de010178  -> (null) 
      +0x048 ProbeCount       : 0
   +0x5e0 Packets          : [3] 0x0000029e`ddca8c10 QUIC_PACKET_SPACE
      +0x000 EncryptLevel     : 0 ( QUIC_ENCRYPT_LEVEL_INITIAL )
      +0x004 DeferredDatagramsCount : 0 ''
      +0x008 NextRecvPacketNumber : 0
      +0x010 Connection       : 0x0000029e`de00fbb0 QUIC_CONNECTION
      +0x018 DeferredDatagrams : (null) 
      +0x020 AckTracker       : QUIC_ACK_TRACKER
         +0x000 PacketNumbersReceived : QUIC_RANGE
         +0x018 PacketNumbersToAck : QUIC_RANGE
         +0x030 LargestPacketNumberAcknowledged : 0
         +0x038 LargestPacketNumberRecvTime : 0
         +0x040 AckElicitingPacketsToAcknowledge : 0
      +0x068 WriteKeyPhaseStartPacketNumber : 0
      +0x070 ReadKeyPhaseStartPacketNumber : 0
      +0x078 CurrentKeyPhaseBytesSent : 0
      +0x080 CurrentKeyPhase  : 0y0
      +0x080 AwaitingKeyPhaseConfirmation : 0y0
   +0x5f8 Crypto           : QUIC_CRYPTO
      +0x000 Initialized      : 0y1
      +0x000 InRecovery       : 0y0
      +0x000 TlsDataPending   : 0y0
      +0x000 TlsCallPending   : 0y0
      +0x008 TLS              : 0x0000029e`dd96c8c0 QUIC_TLS
         +0x000 IsServer         : 0y0
         +0x000 TicketReady      : 0y0
         +0x004 LastMessageType  : 1 ( QUIC_TLS_MESSAGE_CLIENT_INITIAL )
         +0x008 TlsSession       : 0x0000029e`dda63310 QUIC_TLS_SESSION
         +0x010 SecConfig        : 0x0000029e`dd8af4b0 QUIC_SEC_CONFIG
         +0x018 Connection       : 0x0000029e`de00fbb0 QUIC_CONNECTION
         +0x020 ReceiveTPCallback : 0x00007ff9`865311ae           unsigned char  msquic!ILT+425(QuicConnReceiveTP)+0
         +0x028 SNI              : 0x0000029e`dd937320  "127.0.0.1"
         +0x030 LocalTPBuffer    : 0x0000029e`ddcabd80  ""
         +0x038 LocalTPLength    : 0x49
      +0x010 TlsState         : QUIC_TLS_PROCESS_STATE
         +0x000 HandshakeComplete : 0y0
         +0x000 SessionResumed   : 0y0
         +0x000 EarlyDataAttempted : 0y1
         +0x000 EarlyDataAccepted : 0y0
         +0x004 ReadKey          : 0 ( QUIC_PACKET_KEY_INITIAL )
         +0x008 WriteKey         : 1 ( QUIC_PACKET_KEY_0_RTT )
         +0x00c AlertCode        : 0
         +0x00e BufferLength     : 0x9c
         +0x010 BufferAllocLength : 0x1000
         +0x014 BufferTotalLength : 0x9c
         +0x018 BufferOffsetHandshake : 0
         +0x01c BufferOffset1Rtt : 0
         +0x020 Buffer           : 0x0000029e`de0edc40  "???"
         +0x028 ReadKeys         : [6] 0x0000029e`ddcfac90 QUIC_PACKET_KEY
         +0x058 WriteKeys        : [6] 0x0000029e`de384db0 QUIC_PACKET_KEY
      +0x098 MaxSentLength    : 0
      +0x09c UnAckedOffset    : 0
      +0x0a0 NextSendOffset   : 0
      +0x0a4 RecoveryNextOffset : 0
      +0x0a8 RecoveryEndOffset : 0
      +0x0b0 SparseAckRanges  : QUIC_RANGE
         +0x000 SubRanges        : 0x0000029e`ddf5c940 QUIC_SUBRANGE
         +0x008 UsedLength       : 0
         +0x00c AllocLength      : 8
         +0x010 MaxAllocSize     : 0x100000
      +0x0c8 RecvTotalConsumed : 0
      +0x0cc RecvEncryptLevelStartOffset : 0
      +0x0d0 RecvBuffer       : QUIC_RECV_BUFFER
         +0x000 CopyOnDrain      : 0y1
         +0x000 ExternalBufferReference : 0y0
         +0x008 OldBuffer        : (null) 
         +0x010 Buffer           : 0x0000029e`ddeee5c0  ".???"
         +0x018 AllocBufferLength : 0x1000
         +0x01c VirtualBufferLength : 0x4000
         +0x020 BaseOffset       : 0
         +0x028 BufferStart      : 0
         +0x030 WrittenRanges    : QUIC_RANGE
   +0x710 Send             : QUIC_SEND
      +0x000 FlushOperationPending : 0y0
      +0x000 DelayedAckTimerActive : 0y0
      +0x000 LastFlushTimeValid : 0y0
      +0x000 TailLossProbeNeeded : 0y0
      +0x008 NextPacketNumber : 0
      +0x010 LastFlushTime    : 0
      +0x018 MaxData          : 0x1000000
      +0x020 PeerMaxData      : 0x1000000
      +0x028 OrderedStreamBytesReceived : 0
      +0x030 OrderedStreamBytesSent : 0
      +0x038 SendFlags        : 0
      +0x040 SendStreams      : QUIC_LIST_ENTRY
         +0x000 Flink            : 0x0000029e`de010300 QUIC_LIST_ENTRY
         +0x008 Blink            : 0x0000029e`de010300 QUIC_LIST_ENTRY
      +0x050 InitialToken     : (null) 
      +0x058 InitialTokenLength : 0
   +0x770 SendBuffer       : QUIC_SEND_BUFFER
      +0x000 PostedBytes      : 0
      +0x008 BufferedBytes    : 0
      +0x010 IdealBytes       : 0x20000
   +0x788 ClientCallbackHandler : 0x00007ff6`79661546     HRESULT  spinquic!ILT+1345(?SpinQuicHandleConnectionEventYAJPEAUQUIC_HANDLEPEAXPEAUQUIC_CONNECTION_EVENTZ)+0
   +0x790 Stats            : QUIC_CONN_STATS
      +0x000 CorrelationId    : 0xf2eab
      +0x008 VersionNegotiation : 0y0
      +0x008 StatelessRetry   : 0y0
      +0x008 ResumptionAttempted : 0y0
      +0x008 ResumptionSucceeded : 0y0
      +0x00c QuicVersion      : 0x190000ff
      +0x010 Timing           : <unnamed-tag>
         +0x000 Start            : 0x1e2dc332
         +0x008 InitialFlightEnd : 0
         +0x010 HandshakeFlightEnd : 0
      +0x028 Schedule         : <unnamed-tag>
         +0x000 LastQueueTime    : 0x1e2dc244
         +0x008 DrainCount       : 2
         +0x010 OperationCount   : 3
      +0x040 Handshake        : <unnamed-tag>
         +0x000 ClientFlight1Bytes : 0
         +0x004 ServerFlight1Bytes : 0
         +0x008 ClientFlight2Bytes : 0
      +0x050 Send             : <unnamed-tag>
         +0x000 TotalPackets     : 0
         +0x008 RetransmittablePackets : 0
         +0x010 SuspectedLostPackets : 0
         +0x018 SpuriousLostPackets : 0
         +0x020 TotalBytes       : 0
         +0x028 TotalStreamBytes : 0
         +0x030 CongestionCount  : 0
         +0x034 PersistentCongestionCount : 0
      +0x088 Recv             : <unnamed-tag>
         +0x000 TotalPackets     : 0
         +0x008 ReorderedPackets : 0
         +0x010 DroppedPackets   : 0
         +0x018 DuplicatePackets : 0
         +0x020 DecryptionFailures : 0
         +0x028 TotalBytes       : 0
         +0x030 TotalStreamBytes : 0
      +0x0c0 Misc             : <unnamed-tag>
         +0x000 KeyUpdateCount   : 0
   +0x858 TestTransportParameter : QUIC_PRIVATE_TRANSPORT_PARAMETER
      +0x000 Type             : 0
      +0x002 Length           : 0
      +0x008 Buffer           : (null)

Memory Leak in QuicPacketBuilderFinalize

While looking at the code, I noticed there are several failure paths in the QuicPacketBuilderFinalize function that don't result in the current packet metadata being cleaned up. It could have a reference to a stream, which doesn't get released. This will at best leak additional memory, and at worst cause a hang in clean up.

spinquic Times Out on Linux Occassionally

It seems that occasionally the Azure Pipelines run of spinquic on Linux is timing out. So far, 100% of the time a rerun doesn't repro the issue. We need to get logging enabled or a core dump on timeout to debug what is going on.

Build Test Code for Kernel Mode

The test code needs to be built for kernel mode and onboarded to AZP.

Package Files for Windows OS Ingestion

All the necessary files for the Windows OS repo need to be packaged up.

Draft 25 Support

https://tools.ietf.org/rfcdiff?url2=draft-ietf-quic-transport-25.txt
https://tools.ietf.org/rfcdiff?url2=draft-ietf-quic-tls-25.txt

Implement the latest draft version.

Build Windows Kernel Mode in AZP

Onboard Windows kernel mode builds to Azure Pipelines.

CidUpdate Test: Client.ForceCidUpdate() failed, 0x8007139f

Describe the bug

Rarely, the CidUpdate test fails with the following error:

D:\a\1\s\src\test\lib\QuicTest.cpp:2135
Client.ForceCidUpdate() failed, 0x8007139f

This is because it's possible the handshake hasn't been confirmed yet. If not, it's expected to return this error code.

Example failure

https://microsoft.visualstudio.com/OS/_build/results?buildId=21601854&view=ms.vss-test-web.build-test-results-tab&runId=123525462&resultId=100825&paneView=debug

Assert on send.c:374 - Builder->Metadata->FrameCount < 8

During an Azure Pipelines run the tests crashed with an assert in send.c:374 for

Builder->Metadata->FrameCount < 8.

Azure Pipelines artifacts (windows-mitls):
https://microsoft.visualstudio.com/OS/_build/results?buildId=21193039&view=artifacts&type=publishedArtifacts

An excerpt from the logs:

[ 0|1454|1610] 39488.137 [conn][02241] Created 1FFAAC279D0, Server=0, CorrelationId=2240
[ 1|1454|0F94] 39488.144 [bind][01075] Cleaning up
[ 0|1454|1610] 39488.145 [conn][02241] New Destination CID: BE5DDC227D15DFD8
[ 0|1454|1610] 39488.152 [conn][02241] Initialize complete
[ 0|1454|1610] 39488.154 [conn][02241] Registered with session: 1FFAA24B010
[ 0|1454|1610] 39488.156 [conn][02241] Assigned worker: 1FFA9D90F40
[ 1|1454|0F94] 39488.186 [bind][01075] Destroyed
[ 1|1454|0F94] 39488.188 [conn][02239] Destroyed
[ 1|1454|13F0] 39488.350 [strm][04731] Created 1FFAA282E70, Connection=1FFAAC279D0 ID=0 IsLocal=1 IsServer=0 IsUniDir=0
[ 1|1454|13F0] 39488.351 [strm][04731] Send State: STARTED
[ 1|1454|13F0] 39488.352 [strm][04731] Recv State: STARTED
[ 1|1454|13F0] 39500.458 [conn][02241] New Remote IP: 127.0.0.1:65142
[ 1|1454|13F0] 39500.791 [bind][01076] Created 1FFAAC4BE60, Udp=1FFAA28FA10 LocalAddr=127.0.0.1:65144 RemoteAddr=127.0.0.1:65142
[ 1|1454|13F0] 39500.794 [conn][02241] New Source CID: null
[ 1|1454|13F0] 39500.796 [conn][02241] New Local IP: 127.0.0.1:65144
[ 1|1454|13F0] 39501.755 [conn][02241] Version: 0x190000ff
[ 1|1454|13F0] 39501.756 [conn][02241] Handshake start
[ 0|1454|0BE8] 39562.065 [conn][02242] Created 1FFAA2971C0, Server=1, CorrelationId=2241
[ 0|1454|0BE8] 39562.071 [conn][02242] Version: 0x190000ff
[ 0|1454|0BE8] 39562.072 [conn][02242] New Local IP: 127.0.0.1:65142
[ 0|1454|0BE8] 39562.072 [conn][02242] New Remote IP: 127.0.0.1:65144
[ 0|1454|0BE8] 39562.078 [conn][02242] New Destination CID: null
[ 0|1454|0BE8] 39562.080 [conn][02242] New Source CID: BE5DDC227D15DFD8
[ 0|1454|0BE8] 39562.091 [conn][02242] Assigned worker: 1FFA9D949A0
[ 1|1454|11C8] 39562.548 [conn][02242] Initialize complete
[ 1|1454|11C8] 39562.567 [conn][02242] Registered with session: 1FFAA24B010
[ 1|1454|11C8] 39562.677 [conn][02242] New Source CID: BE04B8B31E4557CF
[ 1|1454|11C8] 39562.689 [conn][02242] Handshake start
[ 1|1454|11C8] 39566.521 [conn][02242] Assigned worker: 1FFA9D90D00
[ 0|1454|0F94] 40679.855 [strm][04732] Created 1FFAAC20330, Connection=1FFAA2971C0 ID=0 IsLocal=0 IsServer=0 IsUniDir=0
[ 0|1454|0F94] 40679.857 [strm][04732] Send State: STARTED
[ 0|1454|0F94] 40679.857 [strm][04732] Recv State: STARTED
[ 0|1454|13F0] 40681.046 [conn][02241] Removed Destination CID: BE5DDC227D15DFD8
[ 0|1454|13F0] 40681.047 [conn][02241] New Destination CID: BE04B8B31E4557CF
[ 1|1454|0F94] 40683.290 [conn][02240] Shutdown Complete, PeerFailedToAcknowledged=true
[ 1|1454|0F94] 40683.295 [conn][02240] Unregistered from session: 1FFAA24B010
[ 1|1454|0F94] 40683.296 [conn][02240] Handle closed
[ 1|1454|0F94] 40683.493 [conn][02240] Destroyed
[ 0|1454|13F0] 41196.106 [conn][02241] Handshake complete
[ 0|1454|13F0] 41196.616 [ lib] ASSERT, D:\a\1\s\src\core\send.c:374 - Builder->Metadata->FrameCount < 8

Stateless Retry isn't Consistently Sent

Describe the bug

During draft-27 interop testing, when a quant client connects to our retry endpoint, it doesn't seem to get a RETRY packet in response. I have occasionally seen quicinterop.exe also report a failure of the stateless retry test occasionally.

Steps to reproduce the behavior

Run quicinterop.exe -target:msquic
View output
Expect to see the S included in the passing test cases.

Expected vs actual behavior

It seems we aren't reliably triggering the RETRY packet when configured to always do it. Likely a recent regression.

0-RTT Data isn't Retransmitted After Retry

0-RTT data isn't being retransmitted after the client receives a Retry packet from the server. It should.

Use Universal CRT on Windows

We should update the build allow for using the universal CRT. Some details on that can be found here:

https://docs.microsoft.com/en-us/cpp/porting/upgrade-your-code-to-the-universal-crt?view=vs-2019

OpenSSL Execution is Broken

Describe the bug

Try to run the tests or any tools with OpenSSL and they crash.

Steps to reproduce the behavior

./scripts/build.ps1 -Tls openssl
artifacts/linux/x64_Debug_openssl/quicinterop -target:msquic -test:1

Expected vs actual behavior

> artifacts/linux/x64_Debug_openssl/quicinterop -target:msquic
ASAN:DEADLYSIGNAL
=================================================================
==7018==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7fc895d942b3 bp 0x7fc8861e5ab0 sp 0x7fc8861e5a98 T26)
==7018==The signal is caused by a READ memory access.
==7018==Hint: address points to the zero page.
    #0 0x7fc895d942b2 in SSL_CTX_set_quic_method (/mnt/g/msquic/bld/linux/x64_openssl/openssl/lib/libssl.so.3+0x392b2)
    #1 0x7fc897ccdbc7 in QuicTlsClientSecConfigCreate /mnt/g/msquic/src/platform/tls_openssl.c:923
    #2 0x7fc897c13f67 in QuicConnHandshakeConfigure /mnt/g/msquic/src/core/connection.c:1868
    #3 0x7fc897c14397 in QuicConnInitializeCrypto /mnt/g/msquic/src/core/connection.c:1726
    #4 0x7fc897c14f80 in QuicConnStart /mnt/g/msquic/src/core/connection.c:1656
    #5 0x7fc897c25bf1 in QuicConnProcessApiOperation /mnt/g/msquic/src/core/connection.c:5168
    #6 0x7fc897c26708 in QuicConnDrainOperations /mnt/g/msquic/src/core/connection.c:5323
    #7 0x7fc897bf7b0d in QuicWorkerProcessConnection /mnt/g/msquic/src/core/worker.c:433
    #8 0x7fc897bf8869 in QuicWorkerThread /mnt/g/msquic/src/core/worker.c:510
    #9 0x7fc8973be6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
    #10 0x7fc8970e788e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x12188e)

SEH exception in Drill SetUp & TearDown on Kernel Mode

Describe the bug

msquictest.exe spits out the following when running on kernel mode:

> .\msquictest.exe --kernel --gtest_filter=Drill/WithDrillInitialPacketCidArgs.DrillInitialPacketCids/31
Note: Google Test filter = Drill/WithDrillInitialPacketCidArgs.DrillInitialPacketCids/31
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
Initializing for Kernel Mode tests
[----------] 1 test from Drill/WithDrillInitialPacketCidArgs
unknown file: error: SEH exception with code 0xc0000005 thrown in SetUpTestSuite().
[ RUN      ] Drill/WithDrillInitialPacketCidArgs.DrillInitialPacketCids/31
[       OK ] Drill/WithDrillInitialPacketCidArgs.DrillInitialPacketCids/31 (102 ms)
unknown file: error: SEH exception with code 0xc0000005 thrown in TearDownTestSuite().
[----------] 1 test from Drill/WithDrillInitialPacketCidArgs (104 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (116 ms total)
[  PASSED  ] 1 test.
[  FAILED  ] 0 tests, listed below:

 0 FAILED TESTS
[  FAILED  ] Drill/WithDrillInitialPacketCidArgs: SetUpTestSuite or TearDownTestSuite

Steps to reproduce the behavior

Run msquictest.exe --kernel --gtest_filter=Drill/WithDrillInitialPacketCidArgs.DrillInitialPacketCids/31
View console output

Build Different CPU Architectures on Linux

There is some initial cmake code to build different architectures but it is not implemented for Linux. We need to get x86, x64 and arm working, possibly arm64 (if possible).

Memory Leak in QuicConnHandshakeConfigure

https://microsoft.visualstudio.com/OS/_build/results?buildId=21509070&view=logs&j=54258ef8-e0b4-5486-b151-e5f8227cd852&t=4c87258c-d645-5572-384b-9db650d7b4a5

==3182==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 160 byte(s) in 4 object(s) allocated from:
    #0 0x7fde51da3b50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x7fde51a86961 in QuicAlloc /home/vsts/work/1/s/src/platform/platform_linux.c:111
    #2 0x7fde51a8a3fe in QuicTlsClientSecConfigCreate /home/vsts/work/1/s/src/platform/tls_stub.c:374
    #3 0x7fde51a34f2b in QuicConnHandshakeConfigure /home/vsts/work/1/s/src/core/connection.c:1843
    #4 0x7fde51a350c6 in QuicConnInitializeCrypto /home/vsts/work/1/s/src/core/connection.c:1701
    #5 0x7fde51a35914 in QuicConnStart /home/vsts/work/1/s/src/core/connection.c:1631
    #6 0x7fde51a41f05 in QuicConnProcessApiOperation /home/vsts/work/1/s/src/core/connection.c:5138
    #7 0x7fde51a424d9 in QuicConnDrainOperations /home/vsts/work/1/s/src/core/connection.c:5293
    #8 0x7fde51a22d39 in QuicWorkerProcessConnection /home/vsts/work/1/s/src/core/worker.c:433
    #9 0x7fde51a234a9 in QuicWorkerThread /home/vsts/work/1/s/src/core/worker.c:510
    #10 0x7fde509d16da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)

Now that we've configured ASAN to abort on any errors it finds, including memory leaks, spinquic on Linux regularly fails with the above memory leak.

KeyUpdate Tests on Kernel Mode Fail

Describe the bug

I ran msquic.sys with our internally built tests and all the KeyUpdate tests failed. These failures don't repro when run on msquic.sys that is built internally (which should be identical). The key updates just aren't happening for some reason.

Logs

key_update_0.txt - Max bytes is set, but key update never happens.
key_update_2.txt - Client is supposed to be forcing key update, but I never see logs for it even being tried.

Collect Log Files for Failures

Collecting and uploading all logs for all tests takes a while and generates really big log files. We should only upload log files for failures. That means we would need to be able to collect logs for specific test cases, determine if the test passes or fails, and delete the log file if it passed.

Querying the handshake results

Compare to SslStream.

When the handshake completes the caller may need to know the details of what was negotiated. This information is used for auditing, compatibility checks, and migrations. E.g. "Can I disable this old protocol now or are clients still using it?"

Replace Submodules with CMake Build Instructions

Instead of submoduling all the possible dependencies of msquic, should we instead use CMake to dynamically grab/build/install what ever is needed for the current build configuration? I think it would improve things. It would definitely improve CI time for configurations that don't need to openssl.

MsQuic with OpenSSL Doesn't Load on Different Machine

Currently, when msquic is built using openssl (on Linux) it cannot be loaded on a machine other than the one it was built on. It fails to load the openssl dependencies, even when copied all into the same directory.

Either OpenSSL needs to be statically linked or we need to figure out the steps to package up all dependencies.

New CID Loop per Round Trip

While looking at logs, I saw that sometimes once a round trip new connection IDs were getting created and then retired. After debugging it some, I found that the problem was that the peer who requested the CID to be retired, was then immediately replacing it once it was retired, even though they had already given a replacement. This then triggered the previous new CID to get retired, and thus a loop.

Windows Depends on VS Redist

The default Windows binaries being produced still seem to depend on the VS redistributable. I just deployed them from the latest build (here) and they don't load on my server datacenter VM.

Style Standardization

Should we officially standardize our style (if so, what exactly?) and set up uncrustify in our CI?

It has been suggested that we use camelCase for our variables instead of the current capitalization scheme.

Trouble getting msquic working on ubuntu VM with Chrome, Quic version 25.

I'm currently hitting an issue where msquic is closing the connection fairly early on with an error code of 1 on linux.

Chrome version: Version 82.0.4051.0 (Official Build) canary (64-bit)

Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115807][quic][sock][0x560012b7bcf0] Received [1350] (buflen=[1472]) bytes Src=[131.107.174.191:1568] Dst=[10.0.2.5:443], bind=[0x560012b7bbf0].
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115807][quic][conn][0x7f00f0001150] Path[0] Initialized
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115807][quic][conn][0x7f00f0001150] Queuing 1 UDP datagrams
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][conn][0x7f00f0001150] Recv 1 UDP datagrams
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][conn][0x7f00f0001150] Batch Recv 1 UDP datagrams
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][S][RX][1] LH Ver:0x190000ff DestCid:7255bc44212447f4 SrcCid: Type:I (Token 0 bytes) (Payload 1332 bytes) (PktNum 1 bytes)
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][S][RX][1]   CRYPTO Offset:0 Len:512
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][S][RX][1]   PADDING Len:799
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][conn][0x7f00f0001150] Received 512 crypto bytes, offset=0 Ready=1
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][conn][0x7f00f0001150] App configured max stream count of 100 (type=0).
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][conn][0x7f00f0001150] App configured max stream count of 100 (type=2).
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][list][0x560012cfb340] Indicating NEW_CONNECTION
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][list][0x560012cfb340] App took excessive time (10334 us) in callback.
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][time][0x560012d015f8] Updating Connection 0x7f00f0001150.
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][time][0x560012d015f8] Next Expiration = {324691789578, 0x7f00f0001150}.
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][conn][0x7f00f0001150] Scheduling flags 0x4 to 0x0
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][conn][0x7f00f0001150] Flushing send. Allowance=4050 bytes
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][pktb][0x7f00f0001150] New UDP datagram. Space: 1252
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][pktb][0x7f00f0001150] New QUIC packet. Space: 1252. Type: 0
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][S][TX][0] LH Ver:0x190000ff DestCid: SrcCid:7255bc44212447f4 Type:I (Token 0 bytes) (Payload 24 bytes) (PktNum 4 bytes)
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][S][TX][0]   CONN_CLOSE ErrorCode:0x1 FrameType:0
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][pktb][0x7f00f0001150] Sending batch. 1 datagrams
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][sock][0x560012b7bc50] SocketFd=[176], sending 42 bytes Src=[10.0.2.5:443%2] Dst=[131.107.174.191:1568] (0x7f00f41f3fe0)
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][sock][0x560012b7bc50] Send (0x7f00f41f3fe0) completion succeeded, bytes transferred 42
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][conn][0x7f00f0001150] Flush complete flags=0x0
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][conn][0x7f00f0001150] Abandoning internal, closed connection
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][time][0x560012d015f8] Removing Connection 0x7f00f0001150.
Feb 10 19:42:31 http3UbuntuVm Http3SampleApp: [115811][quic][time][0x560012d015f8] Next Expiration = {NULL}.

This is occurring before we set the SecurityConfig as well. I tried to dive into the code, but it's hard to track back the log message to where it was created.

Crash in QuicLookupRemoveSourceConnectionID

Crash encountered during a test run on Linux:

https://microsoft.visualstudio.com/OS/_build/results?buildId=21495931&view=logs&j=ddae1d7f-4fab-59e5-f384-782422400895&t=452bb8c8-097a-527d-2d8f-f3a86d171860

ASAN:DEADLYSIGNAL
=================================================================
==6487==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7ff72870df12 bp 0x7ff722dfb010 sp 0x7ff722dfb010 T4)
==6487==The signal is caused by a READ memory access.
==6487==Hint: address points to the zero page.
    #0 0x7ff72870df11 in QuicListEntryRemove /home/vsts/work/1/s/src/inc/quic_platform.h:157
    #1 0x7ff72870df11 in QuicHashtableRemove /home/vsts/work/1/s/src/platform/hashtable.c:774
    #2 0x7ff7286e6c1c in QuicLookupRemoveSourceConnectionIDInt /home/vsts/work/1/s/src/core/lookup.c:466
    #3 0x7ff7286e7171 in QuicLookupRemoveSourceConnectionID /home/vsts/work/1/s/src/core/lookup.c:581
    #4 0x7ff7286c5b7b in QuicConnRecvPayload /home/vsts/work/1/s/src/core/connection.c:3491
    #5 0x7ff7286c7583 in QuicConnRecvBatch /home/vsts/work/1/s/src/core/connection.c:3796
    #6 0x7ff7286c8815 in QuicConnRecvDatagrams /home/vsts/work/1/s/src/core/connection.c:3996
    #7 0x7ff7286cc49f in QuicConnDrainOperations /home/vsts/work/1/s/src/core/connection.c:5345
    #8 0x7ff7286ab23e in QuicWorkerProcessConnection /home/vsts/work/1/s/src/core/worker.c:433
    #9 0x7ff7286ab8ef in QuicWorkerThread /home/vsts/work/1/s/src/core/worker.c:510
    #10 0x7ff7284606da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
    #11 0x7ff727b9188e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x12188e)

Unfortunately it seems like there was no core dump file created though. @Maximus- any ideas why this kind of crash wouldn't create a core dump?

Improve ALPN support

ALPN support is present but incomplete in the current implementation:

The client needs to be able to provide multiple ALPN values. Today it's restricted to one.
The server needs to be able to read multiple values from the client during the handshake (e.g. when selecting which server certificate to use).
The server either needs to be able to provide multiple values and have the negotiation resolved internally OR it needs to be documented that the server user is responsible for selecting a single protocol from the client's list (or refusing the connection).
The client API uses chars but the server uses bytes. ALPN values are spec'd as opaque bytes and the API should reflect that. Spec: "The precise set of octet values that identifies the protocol."
After the connection is established the client and server both need to be able to query what ALPN value was negotiated.

Build Windows Debugger Extension

Currently the WinDbg extension isn't being built on GitHub (only internally). We need to get this building. Some helpful info:

https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-engine-and-extension-apis

Remove RSS Dependencies from Core Logic

The core QUIC logic shouldn't need to be aware of RSS. Instead, it should simply match the receive CPU.

microsoft / msquic Goto Github PK

msquic's Introduction

Protocol Features

Library Features

Documentation

Contributing

msquic's People

Contributors

Stargazers

Watchers

Forkers

msquic's Issues

Describe the bug

Steps to reproduce the behavior

Expected vs actual behavior

Client scenarios:

Server scenarios:

Describe the bug

Example failure

Describe the bug

Steps to reproduce the behavior

Expected vs actual behavior

Describe the bug

Steps to reproduce the behavior

Expected vs actual behavior

Describe the bug

Steps to reproduce the behavior

Describe the bug

Logs

Recommend Projects

Recommend Topics

Recommend Org