⚠ PUBLIC PREVIEW ⚠

This repository currently contains a public preview under active development, intended for early adopters and to gather feedback. While in preview, the API is expected to be unstable and breaking changes will occur. See also the Known Issues section for details.

MixedReality-WebRTC is a collection of libraries to help mixed reality app developers to integrate peer-to-peer real-time audio and video communication into their application and improve their collaborative experience.

• Enables multi-track real-time audio / video / data communication with a remote peer
• Provides an abstracted signaling interface to easily switch implementation
• Exposes an API for C++ and C# to integrate into existing apps
• Provides a set of Unity3D components for rapid prototyping and integration
• Supports Microsoft HoloLens (x86) and Microsoft HoloLens 2 (ARM)
• Allows easy use of Mixed Reality Capture (MRC) to stream the view point of the user for multi-device experiences

MixedReality-WebRTC is part of the collection of repositories developed and maintained by the Mixed Reality Sharing team.

Branch	WebRTC	C++ Library	C# Library	Docs
master	m71
feature/m75	m75

The current up-to-date branch with latest developments is the master branch. Initial support for WebRTC release M75 started on the feature/m75 branch, but is not in a working state yet. See #13 for details.

The official documentation is hosted at https://microsoft.github.io/MixedReality-WebRTC/.

• The User Manual contains a general overview and some tutorials.
  • The Hello, Unity world! tutorial introduces the Unity integration by building a simple chat client.
  • The Hello, C# world! tutorial (under construction) introduces the C# API.
• An API reference is also available for the C# library and the Unity integration.

MixedReality-WebRTC is a set of individual building blocks in the form of C++ and C# libraries building upon each other to deliver a consistent API to C++ and C# developers across its supported platforms, and a set of handful drop-in Unity3D components for easy integration.

The overall architecture is as follow:

MixedReality-WebRTC is currently available for Windows 10 Desktop and UWP, with or without Unity, with planned support for Unity deployment on iOS and Android.

Note - In the following and elsewhere in this repository the term "Win32" is used as a synonym for "Windows Desktop", the historical Windows API for Desktop application development, and in opposition to the "Windows UWP" API. However Microsoft Windows versions older than Windows 10 with Windows SDK 17134 (April 2018 Update, 1803) are not officially supported for this project. In particular, older versions of Windows (Windows 7, Windows 8, etc.) are explicitly not supported.

MixedReality-WebRTC is currently under development, and precompiled binary packages for the project's libraries are not yet available. See the Building MixedReality-WebRTC sections below for compiling those libraries from sources.

This repository follows the Pitchfork Layout in an attempt to standardize its hierarchy:

bin/               # Binary outputs (generated)
build/             # Intermediate build artifacts (generated)
docs/              # Documentation
+ manual/          # User manual
examples/          # Examples of use and sample apps
external/          # Third-party external dependencies (git submodules)
libs/              # Source code for the individual libraries
tests/             # Source code for feature tests
tools/             # Utility scripts
+ build/           # Build scripts
+ ci/              # CI Azure pipelines
+ patches/         # Patches applied by build.ps1

The Microsoft.MixedReality.WebRTC.sln Visual Studio 2019 solution located at the root of the repository contains several projects:

• The native C++ library, which can be compiled:
  • for Windows Desktop with the Microsoft.MixedReality.WebRTC.Native.Win32 project
  • for UWP with the Microsoft.MixedReality.WebRTC.Native.UWP project
• A C++ unit tests project Microsoft.MixedReality.WebRTC.Native.Tests
• The C# library project Microsoft.MixedReality.WebRTC
• A C# unit tests project Microsoft.MixedReality.WebRTC.Tests
• A UWP C# sample app project Microsoft.MixedReality.WebRTC.TestAppUWP based on WPF and XAML which demonstrates audio / video / data communication by mean of a simple video chat app.

Note - Currently due to CI limitations some projects are downgraded to VS 2017, but will be reverted to VS 2019 eventually (see #14).

The MixedReality-WebRTC projects consume their input dependencies as NuGet packages for simplicity, because those are complex to build and take a prohibitive amount of time. The steps below are recommended for most users who need to modify the source code of MixedReality-WebRTC without the need to change the input dependencies.

This section describes the prerequisites to build the MixedReality-WebRTC solution using the prebuilt binary input dependencies (NuGet packages). The solution is available at the root of the repository : Microsoft.MixedReality.WebRTC.sln.

• The NuGet packages for the input dependencies require in total approximately 10 GB of disk space. Those dependencies contain unstripped .lib files much larger than the final compiled DLL libraries, for both the Debug and Release build configurations.

• Due to the Windows path length limit, it is recommended to clone the source repository close to the root, e.g. C:\mr-webrtc\ or similar, as the recursive external dependencies create a deep hierarchy which may otherwise produce paths beyond the OS limit.

• Visual Studio 2019 with the following features:

  • The MSVC v142 - VS 2019 C++ x64/x86 build tools toolchain is required to build the C++17 library of MixedReality-WebRTC. This is installed by default with the Desktop development with C++ workload on Visual Studio 2019.

  • The C# library requires a .NET Standard 2.0 compiler, like the Roslyn compiler available as part of Visual Studio when installing the .NET desktop development workload.

  • The UWP libraries and projects require UWP support from the compiler, available as part of Visual Studio when installing the Universal Windows Platform development workload.

• The Unity integration has been tested on Unity 2018.3.x and 2019.1.x. Versions earlier than 2018.3.x may work but are not officially supported.

1. Check out the repository and its dependencies recursively, preferably close to the root (see prerequisites):

   git clone --recursive https://github.com/microsoft/MixedReality-WebRTC.git C:\mr-webrtc

   Note that this may take some time (> 5 minutes) due to the large number of submodules in the WebRTC UWP SDK repository this repository depends on.

2. Build the MixedReality-WebRTC libraries

   • Open the Microsoft.MixedReality.WebRTC.sln Visual Studio solution located at the root of the repository.
   • Build the solution with F7 or Build > Build Solution

   On successful build, the binaries will be generated in a sub-directory under bin/, and the relevant DLLs will be copied by a post-build script to libs\Microsoft.MixedReality.WebRTC.Unity\Assets\Plugins\ for Unity to consume them.

   Note - Be sure to build the solution before opening any Unity integration project. As part of the build, the libraries are copied to the Plugins directory of the Unity integration. There are already some associated .meta files, which have been committed to the repository, to inform Unity of the platform of each DLL. If the Unity project is opened first, before the DLLs are present, Unity will assume those .meta files are stale and will delete them, and then later will recreate some with a different default config once the DLLs are copied. This leads to errors about modules with duplicate names. See the Importing MixedReality-WebRTC chapter of the "Hello, Unity World!" tutorial for more details.

3. Optionally test the installation

   Test the install by e.g. opening the Unity project at libs\Microsoft.MixedReality.WebRTC.Unity, loading the Assets\Microsoft.MixedReality.WebRTC\Unity.Examples\SimpleVideoChat scene and pressing Play. After a few seconds (depending on the machine) the left media player should display the video feed from the local webcam. The Unity console should also display a message about the WebRTC library being initialized successfully.

   See the Hello, Unity World! tutorial for more details.

Note - Although this is strongly discouraged for most users due to its complexity, a detailed step-by-step tutorial on building from source including the Core input dependencies is also available, see Building from sources.

The current version is a public preview under active development, which contains known issues being addressed:

• Mixed Reality Capture (MRC) currently does not work on HoloLens 2 out of the box. Enabling MRC silently fails, and falls back to a video stream without hologram rendering. This is due to a combination of things:
  • MRC only works up to 1080p (see the Mixed reality capture for developers documentation), but the default resolution of the webcam on HoloLens2 is 2272x1278 (see the Locatable Camera documentation). In order to access different resolutions, one need to use video profiles, which are not currently exposed by the WebRTC UWP SDK project. See this issue for details.
  • MRC requires special permission to record the content of the screen:
    • For shared apps (2D slates), this corresponds to the screenDuplication restricted capability, which cannot be obtained by third-party applications.
    • For exclusive-mode apps (fullscreen), there is no particular UWP capability, but the recorded content is limited to the application's own content.
• HoloLens 2 exhibits performance issues thought to be due to:
  • The missing support (#157) for SIMD-accelerated YUV conversion in WebRTC UWP SDK.
  • The use of the highest available video resolution when opening the webcam with the default video profile. Support for selecting a different video profile is not available yet in WebRTC UWP SDK. See this issue for details.
  • The use by default of the VP8 video codec, which is fairly CPU intensive.
• The Debug config of WebRTC core implementation is knows to exhibit performance issues on most devices, including some higher end PCs. Using the Release config of the core WebRTC implementation usually prevents this issue.
• There is currently no clean C++ API; instead the C API used for C# P/Invoke can be used from C++ code, and opaque handles cast to C++ objects. An actual C++ API will eventually be exposed.

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

MixedReality-WebRTC builds upon the WebRTC implementation provided by Google. Security issues and bugs related to this implementation should be reported to Google.

Security issues and bugs related to MixedReality-WebRTC itself or to WebRTC UWP SDK should be reported privately, via email, to the Microsoft Security Response Center (MSRC) [email protected]. You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Further information, including the MSRC PGP key, can be found in the Security TechCenter.