obs-vkcapture starts capture Steam in-game overlay in Proton games about obs-vkcapture HOT 17 OPEN

A small update to that issue.

Vulkan Loader drops VK_INSTANCE_LAYERS support. So from now you should use VK_LOADER_LAYERS_ENABLE and VK_LOADER_LAYERS_DISABLE instead.

So you have to update your environment settings, e.g. from this: export VK_INSTANCE_LAYERS=VK_LAYER_OBS_vkcapture_64:VK_LAYER_VALVE_steam_overlay_64

To that: export VK_LOADER_LAYERS_ENABLE=VK_LAYER_OBS_vkcapture_64,VK_LAYER_VALVE_steam_overlay_64

Note it now uses comma instead of colon sign and also it supports globs. More info you could find here: https://github.com/KhronosGroup/Vulkan-Loader/blob/master/docs/LoaderInterfaceArchitecture.md

Unlike VK_INSTANCE_LAYERS, the new env seem to be ignoring the ordering.

Tried enable both mangohud and vkcapture as explicit layer. with VK_LAYER_OBS_vkcapture_*,VK_LAYER_MANGOHUD_overlay and VK_LAYER_MANGOHUD_overlay,VK_LAYER_OBS_vkcapture_* and both got the same result.

with VK_LOADER_DEBUG=layer the output look like this.

LAYER:               <Application>
LAYER:                 ||
LAYER:               <Loader>
LAYER:                 ||
LAYER:               VK_LAYER_MANGOHUD_overlay
LAYER:                       Type: Explicit
LAYER:                       Manifest: /home/billli11/.local/share/vulkan/explicit_layer.d/MangoHud.json
LAYER:                       Library:  libMangoHud.so
LAYER:                 ||
LAYER:               VK_LAYER_OBS_vkcapture_64
LAYER:                       Type: Explicit
LAYER:                       Manifest: /home/billli11/.local/share/vulkan/explicit_layer.d/obs_vkcapture_64.json
LAYER:                       Library:  /usr/lib/libVkLayer_obs_vkcapture.so
LAYER:                 ||
LAYER:               <Device>

The only way I can force the ordering is set one layer as implicit and one as explicit.
with OBS_VKCAPTURE=1 and VK_LOADER_LAYERS_ENABLE=VK_LAYER_MANGOHUD_overlay it look like this.

LAYER:               <Application>
LAYER:                 ||
LAYER:               <Loader>
LAYER:                 ||
LAYER:               VK_LAYER_OBS_vkcapture_64
LAYER:                       Type: Implicit
LAYER:                           Disable Env Var:  DISABLE_OBS_VKCAPTURE
LAYER:                       Manifest: /usr/share/vulkan/implicit_layer.d/obs_vkcapture_64.json
LAYER:                       Library:  /usr/lib/libVkLayer_obs_vkcapture.so
LAYER:                 ||
LAYER:               VK_LAYER_MANGOHUD_overlay
LAYER:                       Type: Explicit
LAYER:                       Manifest: /home/billli11/.local/share/vulkan/explicit_layer.d/MangoHud.json
LAYER:                       Library:  libMangoHud.so
LAYER:                 ||
LAYER:               <Device>

Also VK_LOADER_LAYERS_ENABLE seem to have higher priority then enable environment. With both OBS_VKCAPTURE=1 and VK_LOADER_LAYERS_ENABLE=VK_LAYER_OBS_vkcapture_*. The layer will always set to explicit(if you copied the json to explicit_layer.d).

edit: exporting VK_ADD_LAYER_PATH=/usr/share/vulkan/implicit_layer.d:$HOME/.local/share/vulkan/implicit_layer.d seem to also make the layers available as explicit layers.

LAYER:                       Type: Explicit
LAYER:                       Manifest: /usr/share/vulkan/implicit_layer.d/MangoHud.json
LAYER:                       Library:  libMangoHud.so

from obs-vkcapture.

Seem like the vulkan loader behavior have been changed again.

with version 1.3.255

VK_INSTANCE_LAYERS now allow to use implicit layer directly. ( No copying or setting VK_ADD_LAYER_PATH is needed )

if an implicit layer and an explicit layer share a same name, BOTH VK_INSTANCE_LAYERS and VK_LOADER_LAYERS_ENABLE will prefer implicit layer.

So you cannot force an layer to be explicit unless you copy layer's json file to explicit_layer.d and change the name to a unique name.

Currently VK_LOADER_LAYERS_ENABLE won't set any order unless creating a new json with unique name.

VK_INSTANCE_LAYERS still respect the order you set even with implicit layer , using it is the best method for now. Until something change again.

from obs-vkcapture.

Currently also encountering the issue others have referred to wherein Game Capture via obs-vkcapture 1.2.2-1 (using OBS 28.1.2-2) does not show MangoHud overlay but does show the game, if said game is launched via Steam. Worth noting that this isn't an issue with games launched from Heroic Games Launcher from personal experience.

INFO | LAYER:     Inserted device 
layer "VK_LAYER_MANGOHUD_overlay" (/usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/vulkan_exp_layer/libMangoHud.so)
INFO | LAYER:     Inserted device layer "VK_LAYER_OBS_vkcapture_64" (/usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/vulkan_imp_layer/7/libVkLayer_obs_vkcapture.so)
INFO | LAYER:     Failed to find vkGetDeviceProcAddr in layer "/usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/vulkan_imp_layer/5/amdvlk64.so"
LAYER:            vkCreateDevice layer callstack setup to:
LAYER:               <Application>
LAYER:                 ||
LAYER:               <Loader>
LAYER:                 ||
LAYER:               VK_LAYER_OBS_vkcapture_64
LAYER:                       Type: Implicit
LAYER:                           Disable Env Var:  DISABLE_OBS_VKCAPTURE
LAYER:                       Manifest: /usr/lib/pressure-vessel/overrides/share/vulkan/implicit_layer.d/7-x86_64-linux-gnu.json
LAYER:                       Library:  /usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/vulkan_imp_layer/7/libVkLayer_obs_vkcapture.so
LAYER:                 ||
LAYER:               VK_LAYER_MANGOHUD_overlay
LAYER:                       Type: Explicit
LAYER:                       Manifest: /usr/lib/pressure-vessel/overrides/share/vulkan/explicit_layer.d/0-x86_64-linux-gnu.json
LAYER:                       Library:  /usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/vulkan_exp_layer/libMangoHud.so
LAYER:                 ||
LAYER:               <Device>
LAYER:                   Using "AMD Radeon RX 5700 XT (RADV NAVI10)" with driver: "/usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/vulkan/2/libvulkan_radeon.so"

I've tried using these launch options in an attempt to follow the force ordering of Vulkan layers to be similar to @Billli11 by moving MangoHud.json to usr/share/vulkan/explicit_layer.d, setting OBS_VKCAPTURE=1, and disabled Steam fossilize and overlay to prevent them being called as layers, but to no avail.

PROTON_LOG=1 AMD_VULKAN_ICD=RADV VK_LOADER_DEBUG=layer OBS_VKCAPTURE=1 VK_LOADER_LAYERS_ENABLE=VK_LAYER_MANGOHUD_overlay VK_LOADER_LAYERS_DISABLE=VK_LAYER_VALVE_steam_overlay_*,VK_LAYER_VALVE_steam_fossilize_* %command% -dev +fps_max 190

I've tried with just OBS_VKCAPTURE=1 VK_LOADER_LAYERS_ENABLE=VK_LAYER_MANGOHUD_overlay and there is no difference.

Details about my PC from Steam. EndeavourOS PC.txt

If you want to also capture Mangohud. you should set mangohud as Implicit layer and obs_vkcapture as explicit.

So you should set

VK_ADD_LAYER_PATH=/usr/share/vulkan/implicit_layer.d:$HOME/.local/share/vulkan/implicit_layer.d MANGOHUD=1 VK_LOADER_LAYERS_ENABLE="VK_LAYER_OBS_vkcapture_*" %command%

VK_ADD_LAYER_PATH add paths where vulkan layers' json is stored, so you don't need to copy json file any more.

LAYER:            vkCreateInstance layer callstack setup to: 
LAYER:               <Application>
LAYER:                 ||  
LAYER:               <Loader>
LAYER:                 ||  
LAYER:               VK_LAYER_MANGOHUD_overlay
LAYER:                       Type: Implicit
LAYER:                           Disable Env Var:  DISABLE_MANGOHUD
LAYER:                       Manifest: /usr/share/vulkan/implicit_layer.d/MangoHud.json
LAYER:                       Library:  libMangoHud.so
LAYER:                 ||  
LAYER:               VK_LAYER_MESA_device_select
LAYER:                       Type: Implicit
LAYER:                           Disable Env Var:  NODEVICE_SELECT
LAYER:                       Manifest: /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
LAYER:                       Library:  libVkLayer_MESA_device_select.so
LAYER:                 ||  
LAYER:               VK_LAYER_OBS_vkcapture_64
LAYER:                       Type: Explicit
LAYER:                       Manifest: /usr/share/vulkan/implicit_layer.d/obs_vkcapture_64.json
LAYER:                       Library:  /usr/lib/libVkLayer_obs_vkcapture.so
LAYER:                 ||  
LAYER:               <Drivers>

EDIT:

VK_ADD_LAYER_PATH and VK_LOADER_LAYERS_ENABLE also work on steam overlay

the overlay can be set to layer after obs_vkcapture.

LAYER:               <Application>
LAYER:                 ||
LAYER:               <Loader>
LAYER:                 ||
LAYER:               VK_LAYER_OBS_vkcapture_64
LAYER:                       Type: Implicit
LAYER:                           Disable Env Var:  DISABLE_OBS_VKCAPTURE
LAYER:                       Manifest: /usr/lib/pressure-vessel/overrides/share/vulkan/implicit_layer.d/10-x86_64-linux-gnu.json
LAYER:                       Library:  /usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/vulkan_imp_layer/10/libVkLayer_obs_vkcapture.so
LAYER:                 ||
LAYER:               VK_LAYER_MANGOHUD_overlay
LAYER:                       Type: Explicit
LAYER:                       Manifest: /usr/lib/pressure-vessel/overrides/share/vulkan/explicit_layer.d/00.json
LAYER:                       Library:  libMangoHud.so
LAYER:                 ||
LAYER:               VK_LAYER_VALVE_steam_fossilize_64
LAYER:                       Type: Explicit
LAYER:                       Manifest: /usr/lib/pressure-vessel/overrides/share/vulkan/explicit_layer.d/10-x86_64-linux-gnu.json
LAYER:                       Library:  /usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/vulkan_exp_layer/10/libVkLayer_steam_fossilize.so
LAYER:                 ||
LAYER:               VK_LAYER_VALVE_steam_overlay_64
LAYER:                       Type: Explicit
LAYER:                       Manifest: /usr/lib/pressure-vessel/overrides/share/vulkan/explicit_layer.d/12-x86_64-linux-gnu.json
LAYER:                       Library:  /usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/vulkan_exp_layer/12/steamoverlayvulkanlayer.so
LAYER:                 ||
LAYER:               <Device>

from obs-vkcapture.

Use the old VK_INSTANCE_LAYERS:

VK_INSTANCE_LAYERS=VK_LAYER_MANGOHUD_overlay:VK_LAYER_OBS_vkcapture_64:VK_LAYER_VALVE_steam_overlay_64

You also need to make sure to have mangohud as explicit layer (only when using Vulkan loader < 1.3.255).

from obs-vkcapture.

It depends on the load order of Vulkan layers.
Implicit layers have random load order unfortunately: KhronosGroup/Vulkan-Loader#328

You can copy the layer JSON to explicit_layer.d (in /usr/share/vulkan or ~/.local/share/vulkan/) which will make it available as explicit layer and thus you will be able to set the load order.

I'm not sure if you will be able to make Steam enable the overlay, but load the explicit layer instead of the implicit one (that is enabled just by env var).

export VK_INSTANCE_LAYERS=VK_LAYER_OBS_vkcapture_64:VK_LAYER_VALVE_steam_overlay_64

This will load the layers in order you want (steam overlay won't be in capture), but you need to figure out how to stop Steam loading the implicit overlay layer (maybe just remove it from implicit_layer.d).

from obs-vkcapture.

Yeah, moving Steam overlay files from ~/.local/share/implicit_layer.d to ~/.local/share/explicit_layer.ddid the trick. Steam overlay works fine after that. 👍

from obs-vkcapture.

It depends on the load order of Vulkan layers. Implicit layers have random load order unfortunately: KhronosGroup/Vulkan-Loader#328

You can copy the layer JSON to explicit_layer.d (in /usr/share/vulkan or ~/.local/share/vulkan/) which will make it available as explicit layer and thus you will be able to set the load order.

I'm not sure if you will be able to make Steam enable the overlay, but load the explicit layer instead of the implicit one (that is enabled just by env var).

export VK_INSTANCE_LAYERS=VK_LAYER_OBS_vkcapture_64:VK_LAYER_VALVE_steam_overlay_64

This will load the layers in order you want (steam overlay won't be in capture), but you need to figure out how to stop Steam loading the implicit overlay layer (maybe just remove it from implicit_layer.d).

So I tried moving vkBasalt.json into explicit_layer.d/ and launching my game with VK_INSTANCE_LAYERS=VK_LAYER_vkbasalt:VK_LAYER_OBS_vkcapture_64 which leads to vkcapture capturing vkBasalt, kind of... I am missing something since this is what vkcapture captures

when this is what is actually on my screen

from obs-vkcapture.

Can you open new issue with this please? Also what shaders do you use in vkBasalt? It may be sRGB vs RGB issue.

from obs-vkcapture.

Can you open new issue with this please? Also what shaders do you use in vkBasalt? It may be sRGB vs RGB issue.

Ok, I'll open a new issue. The shaders I'm using are the colourfulness, FakeHDR, Curves ReShade shaders + a LUT. This happens in games that use vkBasalt, but I have encountered something similar in a game in which I am not using reshade shaders or vkBasalt at all. The game is Call Of Duty: Black ops II and this is what vkCapture captures:

The image is significantly darker and less saturated than what I see on my screen.

from obs-vkcapture.

I am not sure if my issue is related, but MangoHud flickers in recordings and obs preview when I use this plugin with Elden Ring, this doesn't happen with Dark Souls III tho.

from obs-vkcapture.

A small update to that issue.

Vulkan Loader drops VK_INSTANCE_LAYERS support. So from now you should use VK_LOADER_LAYERS_ENABLE and VK_LOADER_LAYERS_DISABLE instead.

So you have to update your environment settings, e.g. from this:
export VK_INSTANCE_LAYERS=VK_LAYER_OBS_vkcapture_64:VK_LAYER_VALVE_steam_overlay_64

To that:
export VK_LOADER_LAYERS_ENABLE=VK_LAYER_OBS_vkcapture_64,VK_LAYER_VALVE_steam_overlay_64

Note it now uses comma instead of colon sign and also it supports globs. More info you could find here: https://github.com/KhronosGroup/Vulkan-Loader/blob/master/docs/LoaderInterfaceArchitecture.md

from obs-vkcapture.

Currently also encountering the issue others have referred to wherein Game Capture via obs-vkcapture 1.2.2-1 (using OBS 28.1.2-2) does not show MangoHud overlay but does show the game, if said game is launched via Steam. Worth noting that this isn't an issue with games launched from Heroic Games Launcher from personal experience.

INFO | LAYER:     Inserted device 
layer "VK_LAYER_MANGOHUD_overlay" (/usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/vulkan_exp_layer/libMangoHud.so)
INFO | LAYER:     Inserted device layer "VK_LAYER_OBS_vkcapture_64" (/usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/vulkan_imp_layer/7/libVkLayer_obs_vkcapture.so)
INFO | LAYER:     Failed to find vkGetDeviceProcAddr in layer "/usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/vulkan_imp_layer/5/amdvlk64.so"
LAYER:            vkCreateDevice layer callstack setup to:
LAYER:               <Application>
LAYER:                 ||
LAYER:               <Loader>
LAYER:                 ||
LAYER:               VK_LAYER_OBS_vkcapture_64
LAYER:                       Type: Implicit
LAYER:                           Disable Env Var:  DISABLE_OBS_VKCAPTURE
LAYER:                       Manifest: /usr/lib/pressure-vessel/overrides/share/vulkan/implicit_layer.d/7-x86_64-linux-gnu.json
LAYER:                       Library:  /usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/vulkan_imp_layer/7/libVkLayer_obs_vkcapture.so
LAYER:                 ||
LAYER:               VK_LAYER_MANGOHUD_overlay
LAYER:                       Type: Explicit
LAYER:                       Manifest: /usr/lib/pressure-vessel/overrides/share/vulkan/explicit_layer.d/0-x86_64-linux-gnu.json
LAYER:                       Library:  /usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/vulkan_exp_layer/libMangoHud.so
LAYER:                 ||
LAYER:               <Device>
LAYER:                   Using "AMD Radeon RX 5700 XT (RADV NAVI10)" with driver: "/usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/vulkan/2/libvulkan_radeon.so"

I've tried using these launch options in an attempt to follow the force ordering of Vulkan layers to be similar to @Billli11 by moving MangoHud.json to usr/share/vulkan/explicit_layer.d, setting OBS_VKCAPTURE=1, and disabled Steam fossilize and overlay to prevent them being called as layers, but to no avail.

PROTON_LOG=1 AMD_VULKAN_ICD=RADV VK_LOADER_DEBUG=layer OBS_VKCAPTURE=1 VK_LOADER_LAYERS_ENABLE=VK_LAYER_MANGOHUD_overlay VK_LOADER_LAYERS_DISABLE=VK_LAYER_VALVE_steam_overlay_*,VK_LAYER_VALVE_steam_fossilize_* %command% -dev +fps_max 190

I've tried with just OBS_VKCAPTURE=1 VK_LOADER_LAYERS_ENABLE=VK_LAYER_MANGOHUD_overlay and there is no difference.

Details about my PC from Steam.
EndeavourOS PC.txt

from obs-vkcapture.

@Billli11 Thank you for the clarification, it seems I was doing it in reverse. The above command you mentioned works when I set it as launch options on the game I'm running on Steam.

from obs-vkcapture.

So is it possible to achieve the following order, and if so how?
I've got VK_ADD_LAYER_PATH=/usr/share/vulkan/implicit_layer.d:$HOME/.local/share/vulkan/implicit_layer.d and VK_LOADER_LAYERS_ENABLE=.

I would like it to follow:

<Application>
|
<Loader>
|
VK_LAYER_MESA_device_select (implicit)
|
VK_LAYER_MANGOHUD_overlay (implicit)
|
VK_LAYER_OBS_vkcapture_64 (explicit)
|
VK_LAYER_VALVE_steam_overlay_64 (explicit)
|
VK_LAYER_VALVE_steam_fossilize_64 (explicit)
|
<Drivers>

But setting VK_LOADER_LAYERS_ENABLE=VK_LAYER_OBS_vkcapture_*,VK_LAYER_VALVE_steam_overlay_*,VK_LAYER_VALVE_steam_fossilize_* always results in vkcapture getting sandwiched with the overlay layer ending up captured:

<Application>
|
<Loader>
|
VK_LAYER_MESA_device_select (implicit)
|
VK_LAYER_MANGOHUD_overlay (implicit)
|
VK_LAYER_VALVE_steam_overlay_64 (explicit)
|
VK_LAYER_OBS_vkcapture_64 (explicit)
|
VK_LAYER_VALVE_steam_fossilize_64 (explicit)
|
<Drivers>

from obs-vkcapture.

Oh, after reading #14 (comment) I hadn't tried it.

But using your example I'm always getting this order even if I move the entries around.

LAYER:            vkCreateInstance layer callstack setup to:
LAYER:               <Application>
LAYER:                 ||
LAYER:               <Loader>
LAYER:                 ||
LAYER:               VK_LAYER_MESA_device_select
LAYER:                       Type: Implicit
LAYER:                           Disable Env Var:  NODEVICE_SELECT
LAYER:                       Manifest: /usr/lib/pressure-vessel/overrides/share/vulkan/implicit_layer.d/6.json
LAYER:                       Library:  libVkLayer_MESA_device_select.so
LAYER:                 ||
LAYER:               VK_LAYER_VALVE_steam_fossilize_64
LAYER:                       Type: Implicit
LAYER:                           Disable Env Var:  DISABLE_VK_LAYER_VALVE_steam_fossilize_1
LAYER:                       Manifest: /usr/lib/pressure-vessel/overrides/share/vulkan/implicit_layer.d/1-x86_64-linux-gnu.json
LAYER:                       Library:  /usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/vulkan_imp_layer/1/libVkLayer_steam_fossilize.so
LAYER:                 ||
LAYER:               VK_LAYER_MANGOHUD_overlay
LAYER:                       Type: Explicit
LAYER:                       Manifest: /usr/lib/pressure-vessel/overrides/share/vulkan/explicit_layer.d/1-x86_64-linux-gnu.json
LAYER:                       Library:  /usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/vulkan_exp_layer/1/libMangoHud.so
LAYER:                 ||
LAYER:               VK_LAYER_VALVE_steam_overlay_64
LAYER:                       Type: Explicit
LAYER:                       Manifest: /usr/lib/pressure-vessel/overrides/share/vulkan/explicit_layer.d/7-x86_64-linux-gnu.json
LAYER:                       Library:  /usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/vulkan_exp_layer/7/steamoverlayvulkanlayer.so
LAYER:                 ||
LAYER:               VK_LAYER_OBS_vkcapture_64
LAYER:                       Type: Explicit
LAYER:                       Manifest: /usr/lib/pressure-vessel/overrides/share/vulkan/explicit_layer.d/3-x86_64-linux-gnu.json
LAYER:                       Library:  /usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/vulkan_exp_layer/3/libVkLayer_obs_vkcapture.so
LAYER:                 ||
LAYER:               <Drivers>

from obs-vkcapture.

@DonKatsu
Try this

VK_ADD_LAYER_PATH="/usr/share/vulkan/implicit_layer.d:$HOME/.local/share/vulkan/implicit_layer.d" VK_LOADER_LAYERS_ENABLE="VK_LAYER_VALVE_steam_overlay_*" MANGOHUD=1 OBS_VKCAPTURE=1 %command%

mangohud may not be in front of obs-vkcapture but we can no longer forcing the ordering using VK_LOADER_LAYERS_ENABLE

@nowrep

VK_INSTANCE_LAYERS has been drop and no longer work. link

Edit 1: as of version 1.3.245

VK_INSTANCE_LAYERS seem to be working again. And it's respecting VK_ADD_LAYER_PATH, same as VK_LOADER_LAYERS_ENABLE

But currently there are a regression that the ordering of VK_INSTANCE_LAYERS will not be follow (issue). And have been fixed upstream and will be included in 1.3.247

from obs-vkcapture.

@Billli11
Since I was trying to capture mangohud but not the Steam overlay, I guess I'll have to settle with making vkcapture explicit and leaving mangohud and Steam overlay as implicit, since the overlay layer seems to always take priority.
I don't often use the Steam overlay anyway, but it would be ideal to not have it show up in OBS when I do. It's really a shame that newer option doesn't do ordering.

from obs-vkcapture.