microsoft / onnxruntime-unrealengine Goto Github PK

Apply a Style Transfer Neural Network in real time with Unreal Engine 5 leveraging ONNX Runtime.

License: MIT License

C# 5.85% C++ 94.15%

onnxruntime-unrealengine's Introduction

This code sample is to show you how to use the new Neural Network Inference (NNI) Plugin in Unreal Engine 5 which implements ONNX Runtime to allow you to add Machine Learning (ML) Models in your projects. ONNX Runtime is a library to optimize and accelerate machine learning inferencing.

We are using open source models from the ONNX model zoo to apply a style transform to the scene during game play. Check out the before and after pictures below to see how one of the models is able to stylize the scene.

Before:

After:

Statement of Purpose

This repository aims to grow the understanding of using ML models through the NNI Plugin in Unreal Engine 5 by providing an example of implementation and references to support the Microsoft Build conference in 2022. It is not intended to be a released product. Therefore, this repository is not for discussing ONNX Runtime, the NNI plugin or requesting new features.

Prerequisites

Unreal 5
- Check to make sure you have the minimum hardware and software requirements to run UE.
- Installing UE5 can take hours depending on your network and hardware. Please see the UE docs for information on how to install UE5
Visual Studio
- Visual Studio 2019 v16.11.5 or later (recommended)
- Visual Studio 2022
  - Include .Net Core 3.1
Windows
The initial version of the NNI plugin supports CPU inference on PC (Windows/Linux/Mac) and Consoles (PS5/Xbox Series X). At the moment, GPU evaluation is only supported for Windows DirectX 12.
This project will only work up to version 5.2 of Unreal. Check out these docs for how to use the updated NNE plugin

How to run this project

Options 1: Run with Epic Games Launcher

Open the Epic Game Launcher and launch Unreal 5

Browse to the location that you cloned the project.
Select the FPStyleTransfer.uproject file and click open. This will load the same project.

NOTE: If you receieve the below message click Fix Now and it will recompile the project.

Once the project is loaded click play to see the project work!

Option 2: Open and run in Visual Studio.

Browse to the location that you cloned the project.
Open the FPStyleTransfer.sln file.

Hit F5 to run the project.
Click play to see the project work!

How to select a different model

There are 5 differnet models included from the ONNX Model Zoo. You can find them under the Content > Models folder in the UE Content Browser. To add your own models, drag and drop them into the Models folder in the Content Browser.

To select one of the other preloaded models or one that you added yourself.

Select the StyleTransferConfig blueprint from the Outliner
In the Details under the Default section find the Neural Network property and select the dropdown. You will see a list of models available.

Select the model you would like to apply from the dropdown, and then click play to see the new style applied.

Resources

Grab open source models to add to your project:

Connect with ONNX Runtime (ORT) on Social!

onnxruntime-unrealengine's People

Contributors

Stargazers

Watchers

onnxruntime-unrealengine's Issues

Unreal 5.0.3 crashes when I drag and drop .onnx file to Content Browser

Hi,

When I drag and drop a new .onnx model into the Models folder in the Content Browser with the NNI Plugin enabled, UE5 crashes and I receive this error:

LoginId:85534685404c849a1fedc785f6cc6425
EpicAccountId:be4981e3506049728d6de8b73f739cfb

Assertion failed: false [File:D:\build++UE5\Sync\Engine\Plugins\Experimental\NNI\Source\Private\ThirdPartyHelperAndDLLLoader\Private\ORTExceptionHandler.cpp] [Line: 16] ONNXRuntime threw an exception with code 9, e.what(): "Could not find an implementation for Div(1) node with name 'Block386'".

UnrealEditor_ThirdPartyHelperAndDLLLoader
UnrealEditor_NeuralNetworkInference
UnrealEditor_NeuralNetworkInference
UnrealEditor_NeuralNetworkInference
UnrealEditor_NeuralNetworkInference
UnrealEditor_NeuralNetworkInference
UnrealEditor_NeuralNetworkInferenceEditor
UnrealEditor_UnrealEd
UnrealEditor_AssetTools
UnrealEditor_AssetTools
UnrealEditor_UnrealEd
UnrealEditor_UnrealEd
UnrealEditor_UnrealEd
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_UnrealEd
UnrealEditor_UnrealEd
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
kernel32
ntdll

How would I be able to work around this problem? Or am I missing a step?

It only works when ejecting player using F8

Hi, I'm testing this tutorial. But it doesn't work for me.
This is before I push play button.

This is after I push play button.

But it works after I push ejectting player button using F8

I think this happens because output of previous tick is used for input of next style transfer tick.
Anyone knows the reason?

Crash when model load

Hi,

When I drag and drop a my .onnx model into the Models folder in the Content Browser with the NNI Plugin enabled, UE 5.0.3 crashes and I receive this warnings and error:

LoginId:566104db435199706d3e599d90248b6d
EpicAccountId:3615f45d01e442e98192e379821e35a2

Assertion failed: false [File:D:\build\++UE5\Sync\Engine\Plugins\Experimental\NNI\Source\Private\ThirdPartyHelperAndDLLLoader\Private\ORTExceptionHandler.cpp] [Line: 16] ONNXRuntime threw an exception with code 6, e.what(): "Exception during initialization: D:\build\++UE5\Sync\Engine\Plugins\Experimental\NNI\Source\ThirdParty\ORT\Private\Windows\core\providers\dml\DmlExecutionProvider\src\AbiCustomRegistry.cpp(507)\UnrealEditor-ONNXRuntime.dll!00007FFC9E6917DA: (caller: 00007FFC9E69D3D1) Exception(3) tid(7e14) 80070057 The parameter is incorrect. ".

UnrealEditor_ThirdPartyHelperAndDLLLoader!FORTExceptionHandler::ThrowPseudoException() [D:\build\++UE5\Sync\Engine\Plugins\Experimental\NNI\Source\Private\ThirdPartyHelperAndDLLLoader\Private\ORTExceptionHandler.cpp:16]
UnrealEditor_NeuralNetworkInference!Ort::ThrowOnError() [D:\build\++UE5\Sync\Engine\Plugins\Experimental\NNI\Source\ThirdParty\ORT\Internal\core\session\onnxruntime_cxx_inline.imp:17]
UnrealEditor_NeuralNetworkInference!Ort::Session::Session() [D:\build\++UE5\Sync\Engine\Plugins\Experimental\NNI\Source\ThirdParty\ORT\Internal\core\session\onnxruntime_cxx_inline.imp:551]
UnrealEditor_NeuralNetworkInference!UNeuralNetwork::FImplBackEndUEAndORT::Load() [D:\build\++UE5\Sync\Engine\Plugins\Experimental\NNI\Source\NeuralNetworkInference\Private\NeuralNetworkImplBackEndUEAndORT.cpp:349]
UnrealEditor_NeuralNetworkInference!UNeuralNetwork::Load() [D:\build\++UE5\Sync\Engine\Plugins\Experimental\NNI\Source\NeuralNetworkInference\Private\NeuralNetwork.cpp:549]
UnrealEditor_NeuralNetworkInference!UNeuralNetwork::Load() [D:\build\++UE5\Sync\Engine\Plugins\Experimental\NNI\Source\NeuralNetworkInference\Private\NeuralNetwork.cpp:99]
UnrealEditor_NeuralNetworkInferenceEditor!UNeuralNetworkFactory::FactoryCreateFile() [D:\build\++UE5\Sync\Engine\Plugins\Experimental\NNI\Source\NeuralNetworkInferenceEditor\Private\NeuralNetworkFactory.cpp:46]
UnrealEditor_UnrealEd!UFactory::ImportObject() [D:\build\++UE5\Sync\Engine\Source\Editor\UnrealEd\Private\Factories\Factory.cpp:205]
UnrealEditor_AssetTools!UAssetToolsImpl::ImportAssetsInternal() [D:\build\++UE5\Sync\Engine\Source\Developer\AssetTools\Private\AssetTools.cpp:2254]
UnrealEditor_AssetTools!UAssetToolsImpl::ImportAssets() [D:\build\++UE5\Sync\Engine\Source\Developer\AssetTools\Private\AssetTools.cpp:1277]
UnrealEditor_UnrealEd!FImportFilesByPath::Run() [D:\build\++UE5\Sync\Engine\Source\Editor\UnrealEd\Private\Subsystems\ImportSubsystem.cpp:97]
UnrealEditor_UnrealEd!UImportSubsystem::HandleNextTick() [D:\build\++UE5\Sync\Engine\Source\Editor\UnrealEd\Private\Subsystems\ImportSubsystem.cpp:171]
UnrealEditor_UnrealEd!TBaseUObjectMethodDelegateInstance<0,UImportSubsystem,void __cdecl(void),FDefaultDelegateUserPolicy>::Execute() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Delegates\DelegateInstancesImpl.h:594]
UnrealEditor_Engine!FTimerUnifiedDelegate::Execute() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Public\TimerManager.h:51]
UnrealEditor_Engine!FTimerManager::Tick() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\TimerManager.cpp:925]
UnrealEditor_UnrealEd!UEditorEngine::Tick() [D:\build\++UE5\Sync\Engine\Source\Editor\UnrealEd\Private\EditorEngine.cpp:1402]
UnrealEditor_UnrealEd!UUnrealEdEngine::Tick() [D:\build\++UE5\Sync\Engine\Source\Editor\UnrealEd\Private\UnrealEdEngine.cpp:474]
UnrealEditor!FEngineLoop::Tick() [D:\build\++UE5\Sync\Engine\Source\Runtime\Launch\Private\LaunchEngineLoop.cpp:5215]
UnrealEditor!GuardedMain() [D:\build\++UE5\Sync\Engine\Source\Runtime\Launch\Private\Launch.cpp:183]
UnrealEditor!GuardedMainWrapper() [D:\build\++UE5\Sync\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:147]
UnrealEditor!LaunchWindowsStartup() [D:\build\++UE5\Sync\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:283]
UnrealEditor!WinMain() [D:\build\++UE5\Sync\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:330]
UnrealEditor!__scrt_common_main_seh() [d:\a01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288]
kernel32
ntdll

How could I resolve this?

I found similar issues on onnxruntime repo but I'm not sure if these apply to my case.
microsoft/onnxruntime#12677
microsoft/onnxruntime#12761

Unreal Engine Project File Working on Linux Machine Ubuntu 22.04

Hello,

Sorry if this isn't the correct place to ask but in the ReadME it states: "The initial version of the NNI plugin supports CPU inference on PC (Windows/Linux/Mac) and Consoles (PS5/Xbox Series X). At the moment, GPU evaluation is only supported for Windows DirectX 12."

I was just wondering what Linux OS Specs were used to run this project successfully as off the bat a Linux UE5 install will use VSCode and for this particular example project it requires Visual Studio 2019 or higher.

Just wanted more clarification.

Add instructions for installing UE5.
- It took me hours this afternoon to install UE5 on my laptop. I wonder if we should warn users, it would take hours (depending on HW and network, of course) to install UE 5.
- Found a good official doc about installing UE engine. - https://docs.unrealengine.com/5.0/en-US/installing-unreal-engine/ It might be good to link it in ReadMe.
When loading uproject from UE launch, is this expected?
In the fourth step of VS - Click play to see the project work!, suggest to note the "Play" is in UE and it's the play button in the top toolbar.
Suggest add .NET Core 3.1 as requirements since I got the following error in another clean Windows machine when building the project from UE5.

It was not possible to find any compatible framework version
The framework 'Microsoft.NETCore.App', version '3.1.0' (x64) was not found.
  - The following frameworks were found:
      6.0.4 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

You can resolve the problem by installing the specified framework and/or SDK.

The specified framework can be found at:
  - https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=3.1.0&arch=x64&rid=win10-x64```

How to replace the current style transfer model with any 3P model?

Could we add instructions to guide users how to use their own model?