dolbyio / comms-sdk-unreal Goto Github PK

Add jobs to build the plugin using Unreal Engine 5.1 on Windows and macOS.

Describe the feature request

Add an actor which can initialize the plugin and connect to a conference with a simple drag and drop onto the scene to simplify getting started.

Describe the feature request

There should be a way to obtain the ID of the conference we're connected to, preferably as an additional parameter to "On Connected", however this is not backwards-compatible. Alternatively, a simple property or getter should suffice.

Describe the feature request

UE 5.2.0 is now available and the plugin should be built for it. Whether to remove 5.0 to be discussed.

Describe the bug

1. Start some music in background.
2. Set Windows to attenuate sounds by 80% when it detects communication in the sound control panel.
3. Connect to conference.
4. Disconnect from conference.
5. Connect again.
6. Set Windows to do nothing when it detects communication.
7. Music volume is still lower, incorrectly. Possible to completely silence the music, but not to have it at 100% nor 50%. Need to disconnect while "Do nothing" is selected and reconnect to fix.

And more simply: if you quit the conference and reconnect with sound attenuated, you cannot disable the attenuation mid-conference.

Another, related reproduction scenario is:

1. Start some music in background.
2. Set Windows to attenuate sounds by 80% when it detects communication in the sound control panel.
3. Connect to conference.
4. Quit game in editor.
5. Reopen game, don't join conference.
6. Music volume is lowered, incorrectly, even though we are not even in conference.

Expected Behavior

Music volume should never be lowered out of conference.

AND

Either:

• music volume should not be lower when setting "Do nothing" (and should work correctly with other settings as well)
  or
• music volume should never be lower (plugin sound should not be detected as communication by Windows)

Minimalistic code (recommended)

N/A

Specifications

• OS: Windows
• Version:
  • comms-sdk-unreal 1.0.0

Describe the feature request

The Get Texture function does not modify any state and should be marked const to avoid an exec pin in Blueprints. This will however not be backwards-compatible.

Describe the feature request

Consider remaking sample Blueprints in UE4 to make them more backwards-compatible.

The plugin's dynamic library does not contain valid metadata.

Describe the feature request

Sometimes the texture which is supposed to contain remote video ends up being completely white and the following log may be displayed (sometimes the texture is white without this log):

[2023.02.03-11.55.05:367][203]LogOutputDevice: Error: === Handled ensure: ===
[2023.02.03-11.55.05:368][203]LogOutputDevice: Error:
[2023.02.03-11.55.05:369][203]LogOutputDevice: Error: Ensure condition failed: oldValue == newValue [File:D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\HAL\ThreadingBase.cpp] [Line: 300]
[2023.02.03-11.55.05:369][203]LogOutputDevice: Error: oldValue(1) newValue(0) If this check fails make sure that there is a FTaskTagScope(ETaskTag::EParallelRenderingThread) as deep as possible on the current callstack, you can see the current value in ActiveNamedThreads(1), GRenderingThread(393f26c0), GIsRenderingThreadSuspended(0)
[2023.02.03-11.55.05:369][203]LogOutputDevice: Error: Stack:
[2023.02.03-11.55.05:369][203]LogOutputDevice: Error: [Callstack] 0x00007ff63e6c9b5c ReleaseTest50.exe!IsInParallelRenderingThread() []
[2023.02.03-11.55.05:370][203]LogOutputDevice: Error: [Callstack] 0x00007ff643fe1aa9 ReleaseTest50.exe!UTexture::GetResource() []
[2023.02.03-11.55.05:370][203]LogOutputDevice: Error: [Callstack] 0x00007ff643fe309e ReleaseTest50.exe!UTexture::HasPendingRenderResourceInitialization() []
[2023.02.03-11.55.05:370][203]LogOutputDevice: Error: [Callstack] 0x00007ff643f3d74c ReleaseTest50.exe!UStreamableRenderAsset::HasPendingInitOrStreaming() []
[2023.02.03-11.55.05:371][203]LogOutputDevice: Error: [Callstack] 0x00007ff643f52448 ReleaseTest50.exe!UStreamableRenderAsset::WaitForPendingInitOrStreaming() []
[2023.02.03-11.55.05:371][203]LogOutputDevice: Error: [Callstack] 0x00007ff643ff2924 ReleaseTest50.exe!UTexture2D::UpdateResource() []
[2023.02.03-11.55.05:371][203]LogOutputDevice: Error: [Callstack] 0x00007ff63d207e13 ReleaseTest50.exe!DolbyIO::FVideoSink::handle_frame() [C:\Users\kuba\Documents\Unreal Projects\ReleaseTest50\Plugins\DolbyIO_UnrealPlugin_5.0_windows64\Source\Private\DolbyIOVideoSink.cpp:118]
[2023.02.03-11.55.05:371][203]LogOutputDevice: Error: [Callstack] 0x00007ffa8027def5 dolbyio_comms_media.dll!UnknownFunction []
[2023.02.03-11.55.05:371][203]LogOutputDevice: Error: [Callstack] 0x00007ffa7f577294 dolbyio_comms_media.dll!UnknownFunction []
[2023.02.03-11.55.05:372][203]LogOutputDevice: Error: [Callstack] 0x00007ffa7f8837fb dolbyio_comms_media.dll!UnknownFunction []
[2023.02.03-11.55.05:372][203]LogOutputDevice: Error: [Callstack] 0x00007ffa7f9b11c6 dolbyio_comms_media.dll!UnknownFunction []
[2023.02.03-11.55.05:372][203]LogOutputDevice: Error: [Callstack] 0x00007ffa7fad225b dolbyio_comms_media.dll!UnknownFunction []
[2023.02.03-11.55.05:372][203]LogOutputDevice: Error: [Callstack] 0x00007ffa7f7086c6 dolbyio_comms_media.dll!UnknownFunction []
[2023.02.03-11.55.05:372][203]LogOutputDevice: Error: [Callstack] 0x00007ffa7f56aa6b dolbyio_comms_media.dll!UnknownFunction []
[2023.02.03-11.55.05:373][203]LogOutputDevice: Error: [Callstack] 0x00007ffb068926bd KERNEL32.DLL!UnknownFunction []
[2023.02.03-11.55.05:373][203]LogOutputDevice: Error: [Callstack] 0x00007ffb075edfb8 ntdll.dll!UnknownFunction []
[2023.02.03-11.55.05:373][203]LogOutputDevice: Error:
[2023.02.03-11.55.05:373][203]LogStats:                SubmitErrorReport -  0.000 s
[2023.02.03-11.55.05:437][208]LogRHI: Error: Breadcrumbs 'RHIThread'

Reproduction rate is low in Editor but high in packaged games (perhaps slightly higher in UE5).

Tested on Windows, macOS not tested yet (probably same issue though, with perhaps different reproduction rate).

Today, with the current version of the Unreal plugin, it is required to leave (disconnect) from a conference before leaving a map (or level) in Unreal, then reconnect when loading the new map. Otherwise the SDK crashes and the unreal editor / game crashes as well.

If you call the blueprint node "Enable Video" while any other application is gathering the camera input the editor or the build crashes with a fatal error.

It would be good to check first if the camera is available and return false if it couldnt be enabled.

Specifications

• OS: Windows
• Version:
  • comms-sdk-unreal 1.0.0]

Describe the bug

Game may crash when the remote video resolution changes. Probably some race condition between RecreateIfNecessary and UpdateTextureRegions in FVideoSink::FFrame. Happens in UE 5.1.1 and 5.0.3. May not happen in 4.27 (not tested).

Expected Behavior

Resolution changes in remote video happen naturally and do not crash the game.

Minimalistic code (recommended)

N/A

Specifications

• OS: Windows 11 (macOS not tested)
• Version:
  • branches: handle_frame_fix, develop, most likely main and others

Libraries affected:

1. UE4Editor-DolbyIO.dylib
2. libdvclient.dylib
3. libdolbyio_comms_media.dylib

It would be useful to provide async functions (like the current "Get Dolby.io Token"), primarily to simplify initialization and connection and possibly hide the Dolby.io subsystem altogether.

Describe the feature request

Add support for Linux, preferably for the recommended UE 5 distro - Ubuntu 22.04. Provide binaries if possible.

Describe the feature request

It should be possible for the user to see their own video (from camera or screenshare). For camera video, it should be possible to do so without being connected (true preview).

When running the build command locally, e.g.:

"D:\Epic Games\UE_4.27\Engine\Build\BatchFiles\RunUAT.bat"
BuildPlugin
-Rocket
-Plugin="C:\Users\jakub\Documents\Unreal Projects\comms-sdk-unreal\DolbyIO\DolbyIO.uplugin"
-TargetPlatforms=Win64
-Package="C:\Users\jakub\Documents\Unreal Projects\comms-sdk-unreal\Build\DolbyIO"
-StrictIncludes
-VS2019
-NoTurnKeyVariables

The Build/DolbyIO/Binaries/Win64 folder contains the necessary DLLs:

avutil-56.dll
dolbyio_comms_media.dll
dolbyio_comms_sdk.dll
dvclient.dll

These files are not available in the GitHub release packages for some reason, even though the command looks the same. This results in broken releases (which v1.0.0-beta.2 is).

Also there is a zip within a zip - both issues apply to both Windows packages (4.27 and 5.0).

Currently the "supported environments" section can be confusing as it states that we support UE 4.27 and 5.0.3, Windows 10 and macOS 12. It would be more applicable to mention that the supported operating systems for the plugin are:

• Windows 10+
• macOS:
  • UE 4.27: latest Mojave
  • UE 5: latest Monterey

which are determined as being the highest common factor between UE and the C++ SDK (UE4 supports Windows 7, but SDK does not; SDK supports Mojave, but UE5 does not).

And that we support the following Unreal Engine versions: 4.27, 5.0, (soon) 5.1.

Describe the bug

Game crashes when it is closed while a video is being rendered using UE 5.1.1. This usually happens when closing the game for the second time from the editor. The first time is usually (always?) OK.

Expected Behavior

Game closes cleanly.

Minimalistic code (recommended)

N/A

Specifications

• OS: Windows 11 for sure, macOS not tested
• Version:
  • comms-sdk-unreal 1.0.0

Initializing (creating) the SDK is a blocking operation and takes a while, delaying startup or causing freezes if done mid-game. Off-load this operation to another thread to free up the caller and eliminate delays and freezes.

Use the C++ SDK's screensharing capabilities when available. Also involves audiosharing if possible.

Describe the feature request

Currently there can only be one video track per participant, but a screenshare is also a video track and may conflict with their camera feed. Certainly it not currently possible to have a user both stream their camera and their screen, but it should be. If possible, also make it future-proof and allow multiple camera tracks. Also there should be a clear distinction between screenshare and camera tracks (e.g. via an enum or a different event altogether).

Describe the feature request

Using the Get Texture function does its job but requires too much work from the end user. It would be more convenient to have a function such as BindMaterial(Material, Participant) which would be a one-time call to connect a material to a participant's video and would only require putting that material onto something (also a one-time operation), instead of requiring the user to set up a material manually and repeatedly call Get Texture (likely On Tick).

Describe the bug

Unreal Editor deadlocks if the game is closed while any of the UE windows is being screenshared.

Expected Behavior

Game closes cleanly and editor continues normal operation.

Minimalistic code (recommended)

N/A

Specifications

• OS: Windows 11
• Version:
  • comms-sdk-unreal 1.1.0-beta.6

Describe the bug

There is a crash in SDK deallocator code when the editor or a standalone game is closed if the DolbyIO module was ever loaded. This crash is mostly only annoying in the debugger, because otherwise apps segfault on exit and the user does not see it.

Expected Behavior

Editor and game close cleanly.

Minimalistic code (recommended)

N/A

Specifications

• OS: Windows 11
• Version:
  • comms-sdk-unreal 1.1.0-beta.6

Describe the feature request

Add a video material created following this tutorial instruction to the plugin itself, then remove this section and replace with references to the plugin's material. Leave a note in the docs that you can still use custom materials if you use the correct texture parameter name.

Describe the feature request

Use the C++ SDK's register_component_version function.

There is no DolbyIO directory containing the plugin in the zip file and instead the directories/files are in the zip's root, which requires the user to create DolbyIO/ directory inside Plugins/ and then extract the zip to Plugins/DolbyIO/.

To do when appropriate SDK is available.

Describe the feature request

Currently errors are being handled internally in the plugin and are not exposed to the user, so unless they look into UE logs, they will not know why some operation failed. Expose information about errors in Blueprint events, particularly in cases which can often fail or when there is an obvious solution.

Currently it is only possible to download artifacts for builds happening after pushing to main. It would be more useful to have artifacts available for all builds, even if that means reducing the retention period.

Provide a way to get an access token to authenticate the SDK without having to setup a backend server.
As a non-developer who is building an Unreal experience using visual script, I do not have the expertise and resources to deploy a backend service that would allow my Unreal game to retrieve an access token that is used to authenticate the plugin (and Dolby.io SDK).

When you push to main, GitHub will generate artifacts which, when downloaded on Chrome (and possibly other browsers) contain a zip within a zip. The artifacts should contain the zipped DolbyIO folder right away instead of another zip.

Today, with #57 the audio device selection is done at the OS level. This feature request is to allow the developer to select the input and output devices from a blueprint.

The version in https://github.com/DolbyIO/comms-sdk-unreal/blob/main/DolbyIO/DolbyIO.uplugin#L4 is wrong. It should be 1.0 for now.

Describe the feature request

Some ideas:

• sample planes like the ones mentioned in the tutorials
• auto-generate widgets like comboboxes for screenshare/device selection

Steps to reproduce:

1. Create a call with two participants
2. the first participant enables his video
3. the second participant got a participant added event triggered.

My blueprints:

Describe the bug

There is no sound in the demo conference. This is because the player's position is never set because of some flags missing in code. It is possible to get sound if you first join as an active user to a shared conference, disconnect from that and connect to demo conference.

Expected Behavior

There is sound in the demo conference regardless of previous actions.

Specifications

• OS: any
• Version:
  • comms-sdk-unreal 1.1.0-beta.2 (probably also 1.1.0-beta.1)

The readme is very long and some of the information in it should land on GitHub pages or in a similar doc tool.

It has been noted that handling events (e.g. On Token Needed and On Initialized, which are strongly related) in different Blueprints with different lifecycles leads to issues and can be enhanced. Investigate what we can do to improve the experience.

Describe the feature request

The plugin can send screenshare but it does not handle received screenshare tracks. Make it possible to render screens shared by other users in game.

Replace all instances of "Virtual World" with "Virtual Worlds" in names, descriptions, documentation, etc.

Create a tutorial video showing how to get started with the plugin.

From Marketplace guidelines:
1.2.2.c Sellers are required to include a video URL in the Long Description for animated products and strongly recommend to include a video URL for Blueprint products. The video should be a Featurette/Example video like a commercial for the product, displaying its functionality, and should typically be no longer than 1 or 2 minutes.

Provide documentation with an example showing how to use the plugin from C++ (similar to the Blueprint docs).

Describe the bug

Example reproduction:
On key pressed: Get Screenshare Sources -> On Screenshare Sources Received -> Print String (length of Sources).

Pressing the key for the first time works fine, but pressing it for the second time will print the length twice.

Expected Behavior

Length is printed once on every key press.

Specifications

• OS: any
• Version:
  • comms-sdk-unreal 1.1.0-beta.2 (probably beta.1 as well)