Incompatibility with Steam in-game overlay about obs-vkcapture HOT 24 CLOSED

I can't reproduce it, but I also don't have this game to test it with.

Tried with this game https://store.steampowered.com/app/1210250/Island_Saver/ (one free game I found that uses DXVK - edit: it doesn't). It doesn't crash, but capture also doesn't work - probably because of Steam runtime sandboxing? I'm not really sure.

from obs-vkcapture.

I can't reproduce it, but I also don't have this game to test it with.

Well, to be honest, sometimes it doesn't appear for the first time right after running a game in Steam, but after a short period of time Steam in-game overlay freezes the game. Capture works fine for me in Valheim and in some other games I've tested through Proton with obs-vkcapture (Deep Rock Galactic, Metro 2033 Redux, Grim Dawn). I can test some other games, especially I want to try more native games.

How can I assist in debugging this issue? I think I should try to run games outside of Steam to ensure that problem isn't related to my own setup 🤔

from obs-vkcapture.

Duh, no reason to test games outside of Steam, because this issue is related to Steam overlay 😄

from obs-vkcapture.

So if you disable Steam overlay everything works fine?

There is also another vulkan layer that steam loads: libVkLayer_steam_fossilize.so

from obs-vkcapture.

So if you disable Steam overlay everything works fine?

Well, I try to clarify what I mean the above. Capture works fine and my OBS can record without issues despite Steam overlay is disabled/enabled in Steam preferences, but when I'm trying to open Steam in-game overlay (Shift-Tab by default) it freezes/crashes my game, it appears only if I run a game with obs-vkcapture.

As I note, sometime it doesn't happen, but in 99% cases after opening in-game overlay I have to kill my game.

from obs-vkcapture.

Alright, I understand now.

So all you need to do to trigger the issue is just open the Steam overlay (shift+tab) and most of the time it will freeze/crash? Does it happen also when not capturing (OBS not running)?

Can you test it with this game https://store.steampowered.com/app/1067310/Gravitas/ ? I tried this one now, capture works and steam overlay works too without crash.

from obs-vkcapture.

Also, do you see the Steam overlay in the capture (in those rare cases it works)?

from obs-vkcapture.

So all you need to do to trigger the issue is just open the Steam overlay (shift+tab) and most of the time it will freeze/crash?

Yes, exactly.

Does it happen also when not capturing (OBS not running)?

Yes, it does. I tested with and without OBS running - the same result.

Can you test it with this game

Yes, I will report about this game a bit later today.

do you see the Steam overlay in the capture

No, I don't see Steam overlay in the capture. After my game freezes I see only the last frame or black screen at all.

from obs-vkcapture.

That's odd. Can you get a backtrace when it freezes? gdb -p PID

from obs-vkcapture.

Gravitas with Proton 5.13-6 crashes in the same way as other tested games.

from obs-vkcapture.

Well, I'm not sure what exactly I have to do with gdb, but I tried to run Gravitas with PROTON_DUMP_DEBUG_COMMANDS=1, then I go into /tmp/proton_ogden and tried to run gdb_attach from there.

Here GDB output https://pastebin.com/rLivZjyy when I tried to run and take bt. I'm not sure why, but I see gdb: warning: error finding working directory: No such file or directory in there, but file completion starts from the game root directory. Also I tried to manually specify with file command the game executable to Drop.exe in the root directory and SkyArk/Binaries/Win64/Drop-Win64-Shipping.exe, but it doesn't change anything and it complains about debugging symbols

Reading symbols from SkyArk/Binaries/Win64/Drop-Win64-Shipping.exe...
(No debugging symbols found in SkyArk/Binaries/Win64/Drop-Win64-Shipping.exe)

from obs-vkcapture.

Well, I did some things.

• I tried to run Steam with STEAM_RUNTIME_PREFER_HOST_LIBRARIES=0 STEAM_RUNTIME=1. Without success.
• I tried to run Steam without runtime, but it gives me "appdatacache.cpp (2078) : Assertion Failed: !bSharedKVSymbols" as described here ValveSoftware/steam-for-linux#5014 - will try to run Steam without runtime later.
• I tried to disable hardware acceleration just in case in Steam Settings - Interface. Without success.
• Steam reset doesn't help me as well.

But now I notice some warning messages related to Fossilize process. fossilize_log.txt

from obs-vkcapture.

Pretty interesting, I've noticed before that obs-glcapture doesn't work for me (with obs-glcapture %command% in Steam), glxgears captures fine, but today I discover that it works only if I use LD_PRELOAD="/usr/lib64/libobs_glcapture.so". So it refuses to launch with LD_PRELOAD="${LD_PRELOAD} /usr/lib64/libobs_glcapture.so". I tested that with Terraria, Stardew Valley and Valheim with OpenGL render.

from obs-vkcapture.

Here GDB output https://pastebin.com/rLivZjyy when I tried to run and take bt

Try to type continue here, run the game until it crashes and then get backtrace bt.

I've noticed before that obs-glcapture doesn't work for me (with obs-glcapture %command% in Steam), glxgears captures fine, but today I discover that it works only if I use LD_PRELOAD="/usr/lib64/libobs_glcapture.so".

obs-glcapture does LD_PRELOAD="${LD_PRELOAD}:libobs_glcapture.so" %command%, it doesn't use absolute path.

from obs-vkcapture.

Try to type continue here, run the game until it crashes and then get backtrace bt.

Sadly it doesn't produce backtrace when I tried to attach to a process with gdb_attach. But with gdb_run it running a game without Steam overlay... Wonder if I can force to run a game with Steam overlay in this way.

obs-glcapture does LD_PRELOAD="${LD_PRELOAD}:libobs_glcapture.so" %command%, it doesn't use absolute path.

I meant obs-glcapture seems to conflict with Steam in-game overlay too, or at least something in LD_PRELOAD is broken.

from obs-vkcapture.

First of all, I found today that there's a strange issue with gdb (probably related to glibc-2.33), because I can't attach to a proccess by given PID, but debugging with given binary works fine. Just keep in mind maybe it's a Gentoo or generally Glibc 2.33 related bug, but I'm not sure for now. :/

GAME_DEBUGGER=gdb steam &> logfile helps me take a logfile.

The most interesting part from Valheim log.

[obs-vkcapture] ------------------- vulkan capture freed -------------------
Caught fatal signal - signo:11 code:1 errno:0 addr:0x968
Obtained 13 stack frames.
#0  0x007f31c9a611d0 in funlockfile
#1  0x007f31c93f415b in XQueryExtension
#2  0x007f31cbd30751 in glXSwapBuffers
#3  0x007f31cbd34059 in glXSwapBuffers
#4  0x007f31cbd36e55 in XEventsQueued
#5  0x007f31cad549bd in X11_PumpEvents
#6  0x007f31cacfa23b in SDL_PumpEvents_REAL
#7  0x007f31cacfa2ac in SDL_WaitEventTimeout_REAL
#8  0x007f31ca740686 in PopNativeEvent(void*)
#9  0x007f31ca745400 in InputProcess()
#10 0x007f31ca736950 in PlayerMain(int, char**)
#11 0x007f31c98b77ed in __libc_start_main
#12 0x00000000400569 in _start
Game removed: AppID 892970 "", ProcID 25853 

Full log is here Valheim-crash.log Also I do the same for Deep Rock Galactic - DRG.log and The Talos Principle - Talos.log

from obs-vkcapture.

because I can't attach to a proccess by given PID

Why not? You need to have CAP_SYS_PTRACE or just use sudo.

The most interesting part from Valheim log.

Well the capture is Vulkan, but the backtrace shows glX calls, so that's weird.

from obs-vkcapture.

Why not?

Actually I kinda can attach to a process, but it complains libpthread.so like that.

Error while reading shared library symbols for /lib64/libpthread.so.0:
Cannot find user-level thread for LWP 3509: generic error
0x00007f64dc6f66bf in poll () from /lib64/libc.so.6

And because of that I can't properly debug a game crash with obs-vkcapture by PID. Someone from Gentoo forum confirms that a issue is related to gdb/glibc. Also Arch Linux had the same issue before https://bugs.archlinux.org/task/69657.

Well the capture is Vulkan, but the backtrace shows glX calls, so that's weird.

I've double checked that Valheim uses obs-vkcapture in running options right now.

from obs-vkcapture.

Well well, I figured out two things.

1. After downgrading Mesa to 21.0.3 instead of 21.1.0, now Steam overlay works fine. Git version of Mesa also pretty unstable in this case - at the beginning it works, but after a short period of time it will start crashing.
2. Rise of The Tomb Raider refuses to run even with 21.0.3, and it crashing before actually running. rise.log

Also the game produces a crash dump by itself, but Github doesn't accept to publish it here, so I've uploaded it to my Google Drive: https://drive.google.com/file/d/1gUypdC-mhOTn9PtJnz2zvvVtWDQI5S_j/view?usp=sharing

from obs-vkcapture.

No issues here with Mesa 21.1.0, Linux 5.12.4-zen1 + RX570.

Not sure what is the dump file. If it's a core file, you need to load it with gdb and get a backtrace there: gdb /path/to/binary /path/to/core

from obs-vkcapture.

No issues here with Mesa 21.1.0, Linux 5.12.4-zen1 + RX570.

Hm, I'm using 5.11.21 + RX580. Well, I'll try to run with 5.12 then.

Not sure what is the dump file. If it's a core file

Gdb says "is not a core dump: file format not recognized". Anyway I can read this file with cat for example.

upd: No, 5.12 doesn't fix that problem :/

from obs-vkcapture.

What a hell is going on... Well, okay. After multiple attempts to find a source of my issue, I figured out that games with Steam overlay + obs-vkcapture + Mesa 21.1 and above start crashing only when I'm moving my mouse and in-game overlay in opened. 🤯

Even more strange, if I'm moving my mouse inside a game itself (i.e. controlling camera) it won't crash, but as soon as I open overlay and start moving my mouse cursor - it crashes. 🤯

from obs-vkcapture.

Anyway I can read this file with cat for example.

Right, sorry. I just skipped few first lines of the file and concluded it's only binary.

Anyway, since downgrading Mesa works for you, it doesn't really look like issue in obs-vkcapture. Moreso if it happens also when not capturing, as in that case no code actually runs that could potentially interfere.
I'm sorry I can't really help with debugging wine/proton(steam) apps as I don't have much experience there. So unless you can get backtrace, and we can see it crashes inside obs-vkcapture, I'm out of ideas.

from obs-vkcapture.

Sorry for delay, I could not test that before, and this issue makes me sick to be honest, because the reason of these problems in the past isn't obviously at all. But now...

tl;dr: Now I've just tested obs-studio 27 on Artix with your obs-vkcapture package from AUR and now it works both for Proton (tested with Deep Rock Galactic and Proton-6.9-GE-2) and native Valheim. The capture works and overlay doesn't affect my games anymore with obs-vkcapture enabled. 🎉

Here provide some information just to clarify what I found when I tried to debug this issue.

• the issue wasn't a distro-specific problem, so even on Artix with almost the same setup it failed to work properly with Steam in-game overlay;
• for some reason the issue didn't appear with Proton from Valve (at least 6.3-4);
• when Steam in-game overlay works (until I move my mouse), overlay in Big-picture mode fails, and was completely black from the start.

p.s. Thanks you for this tool, I'm so glad to see someone has finally implemented that. Because OBS on Linux lacks of the "game capture" source, I wasn't happy when I saw my browser from other workspace during loading in the game, and obs-vkcapture make my life easier now. :)

from obs-vkcapture.