I'm looking for ways to set up a pipeline for a big project to show warnings when unused assets are being pushed to repository.
This plugin's Python API is really useful for that, however when running on the actual project that has around 200-300 GB of assets (which is why we're trying to reduce it) both the editor and the commandlet crash with "Out of memory" trying to allocate 80-100 GB array.
I'm not really sure what percent of assets are loaded at the point of the crash, but I expect that even if i had 128 GB of RAM available, I would still probably run out of it on the next array allocation when more assets are loaded.
Is it possible to introduce some sort of dynamic loading of assets while they're being checked and unloading when the check is done? Or maybe some other way to drastically reduce ram usage?
Fatal error: [File:D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\GenericPlatform\GenericPlatformMemory.cpp] [Line: 241] Ran out of memory allocating 89227504816 bytes with alignment 8. Last error msg: The paging file is too small for this operation to complete..
UnrealEditor_Core!<lambda_a20f43b083803437f2d783b89afde082>::operator()() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\GenericPlatform\GenericPlatformMemory.cpp:242]
UnrealEditor_Core!FGenericPlatformMemory::OnOutOfMemory() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\GenericPlatform\GenericPlatformMemory.cpp:244]
UnrealEditor_Core!FMallocMimalloc::OutOfMemory() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\HAL\MallocMimalloc.h:50]
UnrealEditor_Core!FMallocMimalloc::Realloc() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\HAL\MallocMimalloc.cpp:137]
UnrealEditor_Core!FMemory::Realloc() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\HAL\FMemory.inl:56]
UnrealEditor_ProjectCleaner!TArray<FAssetData,TSizedDefaultAllocator<32> >::ResizeTo() [D:\RocketSync\5.1.0-23058290+++UE5+Release-5.1\Working\Engine\Source\Runtime\Core\Public\Containers\Array.h:3045]
UnrealEditor_ProjectCleaner!FProjectCleanerDataManager::FindAllAssets() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\ProjectCleaner\Source\ProjectCleaner\Private\Core\ProjectCleanerDataManager.cpp:450]
UnrealEditor_ProjectCleaner!FProjectCleanerDataManager::AnalyzeProject() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\ProjectCleaner\Source\ProjectCleaner\Private\Core\ProjectCleanerDataManager.cpp:56]
UnrealEditor_ProjectCleaner!FProjectCleanerManager::Update() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\ProjectCleaner\Source\ProjectCleaner\Private\Core\ProjectCleanerManager.cpp:38]
UnrealEditor_ProjectCleaner!SProjectCleanerMainUI::OnRefreshBtnClick() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\ProjectCleaner\Source\ProjectCleaner\Private\UI\ProjectCleanerMainUI.cpp:339]
UnrealEditor_ProjectCleaner!TBaseRawMethodDelegateInstance<1,SProjectCleanerMainUI const ,FReply __cdecl(void),FDefaultDelegateUserPolicy>::Execute() [D:\RocketSync\5.1.0-23058290+++UE5+Release-5.1\Working\Engine\Source\Runtime\Core\Public\Delegates\DelegateInstancesImpl.h:459]
UnrealEditor_Slate!SButton::ExecuteOnClick() [D:\build\++UE5\Sync\Engine\Source\Runtime\Slate\Private\Widgets\Input\SButton.cpp:465]
UnrealEditor_Slate!SButton::OnMouseButtonUp() [D:\build\++UE5\Sync\Engine\Source\Runtime\Slate\Private\Widgets\Input\SButton.cpp:390]
UnrealEditor_Slate!<lambda_51e270ca99ea7ce852539efd04dd453f>::operator()() [D:\build\++UE5\Sync\Engine\Source\Runtime\Slate\Private\Framework\Application\SlateApplication.cpp:5046]
UnrealEditor_Slate!FEventRouter::Route<FReply,FEventRouter::FToLeafmostPolicy,FPointerEvent,<lambda_51e270ca99ea7ce852539efd04dd453f> >() [D:\build\++UE5\Sync\Engine\Source\Runtime\Slate\Private\Framework\Application\SlateApplication.cpp:412]
UnrealEditor_Slate!FSlateApplication::RoutePointerUpEvent() [D:\build\++UE5\Sync\Engine\Source\Runtime\Slate\Private\Framework\Application\SlateApplication.cpp:5032]
UnrealEditor_Slate!FSlateApplication::ProcessMouseButtonUpEvent() [D:\build\++UE5\Sync\Engine\Source\Runtime\Slate\Private\Framework\Application\SlateApplication.cpp:5601]
UnrealEditor_Slate!FSlateApplication::OnMouseUp() [D:\build\++UE5\Sync\Engine\Source\Runtime\Slate\Private\Framework\Application\SlateApplication.cpp:5566]
UnrealEditor_ApplicationCore!FWindowsApplication::ProcessDeferredMessage() [D:\build\++UE5\Sync\Engine\Source\Runtime\ApplicationCore\Private\Windows\WindowsApplication.cpp:2219]
UnrealEditor_ApplicationCore!FWindowsApplication::DeferMessage() [D:\build\++UE5\Sync\Engine\Source\Runtime\ApplicationCore\Private\Windows\WindowsApplication.cpp:2726]
UnrealEditor_ApplicationCore!FWindowsApplication::ProcessMessage() [D:\build\++UE5\Sync\Engine\Source\Runtime\ApplicationCore\Private\Windows\WindowsApplication.cpp:1895]
UnrealEditor_ApplicationCore!FWindowsApplication::AppWndProc() [D:\build\++UE5\Sync\Engine\Source\Runtime\ApplicationCore\Private\Windows\WindowsApplication.cpp:925]
user32
user32
InkObj
atlthunk
user32
user32
UnrealEditor_ApplicationCore!FWindowsPlatformApplicationMisc::PumpMessages() [D:\build\++UE5\Sync\Engine\Source\Runtime\ApplicationCore\Private\Windows\WindowsPlatformApplicationMisc.cpp:142]
UnrealEditor!FEngineLoop::Tick() [D:\build\++UE5\Sync\Engine\Source\Runtime\Launch\Private\LaunchEngineLoop.cpp:5293]
UnrealEditor!GuardedMain() [D:\build\++UE5\Sync\Engine\Source\Runtime\Launch\Private\Launch.cpp:202]
UnrealEditor!GuardedMainWrapper() [D:\build\++UE5\Sync\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:107]
UnrealEditor!LaunchWindowsStartup() [D:\build\++UE5\Sync\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:244]
UnrealEditor!WinMain() [D:\build\++UE5\Sync\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:282]
UnrealEditor!__scrt_common_main_seh() [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288]
kernel32
ntdll