galister / wlx-overlay-s Goto Github PK

View Code? Open in Web Editor NEW

78.0 5.0 13.0 1.92 MB

Access your Wayland/X11 desktop from Monado/WiVRn/SteamVR. Now with Vulkan!

License: GNU General Public License v3.0

Rust 100.00%

linux openvr openvr-overlay openxr openxr-overlay steamvr steamvr-overlay vr-overlay wayland x11

wlx-overlay-s's Introduction

WlxOverlay-S

A lightweight OpenXR/OpenVR overlay for Wayland and X11 desktops, inspired by XSOverlay.

WlxOverlay-S allows you to access your desktop screens while in VR.

Compared to similar software, WlxOverlay-S aims to run alongside your other VR games or experiences and have as little performance impact as possible. The UI looks and rendering methods are kept simple and efficient as much as possible.

This is the coming-together of two of my previous projects:

WlxOverlay (SteamVR overlay written in C#)
WlxOverlay-X (OpenXR overlay using StereoKit, written in Rust)

Join the Linux VR Community

We are available on either:

Discord: https://discord.gg/gHwJ2vwSWV
Matrix Space: #linux-vr-adventures:matrix.org

Questions/issues specific to WlxOverlay-S will be handled in the wlxoverlay chat room.

Setup

Grab the latest AppImage from Releases.
chmod +x WlxOverlay-S-*.AppImage
Start Monado, WiVRn or SteamVR.
Run the overlay

AUR package is wlx-overlay-s-git.

You may also want to build from source.

First Start

If you get a screen share pop-up, check the terminal and select the screens in the order it tells you to.

If you selected the screens in the wrong order:

rm ~/.config/wlxoverlay/conf.d/pw_tokens.yaml then restart

SteamVR users: WlxOverlay-S will register itself for auto-start, so you will not need to start it every time.

Please continue reading the guide below.

Getting Started

Working Set

Your working set consists of your currently selected overlays; screens, mirrors, keyboard, etc.

The working set appears in front of you when shown, and can be re-centered by hiding and showing again.

Show and hide your working set using:

Non-vive controller: double-tap B or Y on your left controller.
Vive controller: double-tap the menu button on your left controller (OpenXR, for SteamVR, you might need to bind showhide yourself.)

Pointer Modes AKA Laser Colors

Much of the functionality in WlxOverlay-S depends on what color of laser you are using to interact with a UI element.
Using the default settings, there are 3 modes:

Regular Mode: Blue laser
Right-click Mode: Orange laser
Middle-click Mode: Purple laser

Please see the bindings section below on how to activate these modes.

The guide here uses the colors for ease of getting started.

The Watch

Check your left wrist for the watch. The watch is your primary tool for controlling the app.

The Screens

Hovering a pointer over a screen will move the mouse. If there are more than one pointers hovering a screen, the pointer that was last used to click will take precedence.

The click depends on the laser color:

Blue laser: Left click
Orange laser: Right click
Purple laser: Middle click
Stick up/down: Scroll wheel

To curve screens, grab them with one hand, then with the other hand, hover the laser over the overlay and use the scroll action.

See the bindings section on how to grab, move and resize screens.

The keyboard

The keyboard is fully customizable via the keyboard.yaml file.
Download it into your ~/.config/wlxoverlay/ folder and edit it to your liking.

Typing

Use the BLUE laser when typing regularly.
While using ORANGE laser, all keystrokes will have SHIFT applied.
Purple laser has no effect as of now.

Modifier Keys are sticky. They will remain pressed until you press a non-modifier key, or toggle them off.

Default Bindings

If your bindings are not supported, please reach out.
We would like to work with you and include additional bindings.

Troubleshooting

Check here for tips.

Known Issues

Auto-Start not working on SteamVR 2.4.4

This seems to only affect SteamVR 2.4.4 (stable as of writing). Try switching to temp_v1.27.5 or beta using the SteamVR beta selection if this bothers you. (The linux-v1.14 branch will not work.)

Modifiers get stuck in weird ways

This is a rare issue that can make your desktop not react to click or keys due to a modifier being stuck somewhere. Restarting the overlay fixes this.

Scroll wheel doesn't work

This seems to be an issue specific to Electron apps (Discord, Element, Slack, Spotify) on Wayland. Scrolling will work when using these in your web browser.

X11 limitations

DPI scaling is not supported and may cause the mouse to not follow the laser properly.
Upright screens are not supported and can cause the mouse to act weirdly.
Screen changes (connecting / disconnecting a display, resolution changes, etc) are not handled at runtime. Restart the overlay for these to take effect.

wlx-overlay-s's People

Contributors

Stargazers

Watchers

Forkers

olekolek1000 scrumplex leaphant stuin libjared alexdenerqal havenoftheraven jarettmillard cheesecakecg chibiskuld gurrrrrrett3 kibitob rinlovesyou

wlx-overlay-s's Issues

wlx-overlay-s doesn't show anything inside VR and logs a segfault in libvulkan_radeon.so in dmesg

Description

wlx-overlay-s doesn't show anything on the wrist. When checking dmesg I found the following segfault.

wlx-overlay-s was compiled and executed using the latest commit on main (8116864), from this repo (https://github.com/SpookySkeletons/xr-overlays) with index_camera_passthrough disabled.

The runtime used was Monado + Envision in this commit https://gitlab.com/gabmus/envision/-/pipelines/1172008838

dmesg -T

[dom 11 feb 05:34:02 2024] usb 2-5.1: Found UVC 1.00 device 3D Camera (28de:2400)
[dom 11 feb 05:34:02 2024] input: 3D Camera: eTronVideo as /devices/pci0000:00/0000:00:01.3/0000:02:00.0/usb2/2-5/2-5.1/2-5.1:1.0/input/input28
[dom 11 feb 05:39:58 2024] input: WlxOverlay-S Keyboard-Mouse Hybrid Thing as /devices/virtual/input/input29
[dom 11 feb 05:39:58 2024] show_signal_msg: 53 callbacks suppressed
[dom 11 feb 05:39:58 2024] wlx-overlay-s[27638]: segfault at 10 ip 00007f13cf5cc7ca sp 00007ffc1c151660 error 4 in libvulkan_radeon.so[7f13cf48d000+458000] likely on CPU 5 (core 5, socket 0)
[dom 11 feb 05:39:58 2024] Code: e4 55 89 f5 53 48 81 ec 68 08 00 00 48 89 8c 24 98 00 00 00 b9 40 00 00 00 64 48 8b 04 25 28 00 00 00 48 89 84 24 58 08 00 00 <48> 8b 47 10 48 8d bc 24 50 06 00 00 48 89 bc 24 88 00 00 00 48 89
[dom 11 feb 05:53:26 2024] hrtimer: interrupt took 20489 ns
[dom 11 feb 06:04:21 2024] perf: interrupt took too long (3156 > 3132), lowering kernel.perf_event_max_sample_rate to 63300

System info

My HMD is a Valve Index + Controllers and basestations 2.0

System info from inxi -GSC -xx command:

click to collapse


System:
  Host: localhost.localdomain Kernel: 6.7.4-1-default arch: x86_64 bits: 64
    compiler: gcc v: 13.2.1
  Desktop: KDE Plasma v: 5.27.10 tk: Qt v: 5.15.12 wm: kwin_wayland dm:
    1: GDM 2: SDDM note: stopped Distro: openSUSE Tumbleweed 20240209
CPU:
  Info: 8-core model: AMD Ryzen 7 1700X bits: 64 type: MT MCP arch: Zen rev: 1
    cache: L1: 768 KiB L2: 4 MiB L3: 16 MiB
  Speed (MHz): avg: 3122 high: 3493 min/max: 2200/3400 boost: enabled cores:
    1: 3400 2: 3061 3: 2826 4: 2796 5: 3218 6: 2931 7: 3120 8: 3476 9: 2795
    10: 2811 11: 3400 12: 2843 13: 3493 14: 3450 15: 3098 16: 3242
    bogomips: 108628
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
Graphics:
  Device-1: Blackmagic Design Intensity Pro 4K driver: N/A pcie:
    speed: 2.5 GT/s lanes: 4 bus-ID: 04:00.0 chip-ID: bdbd:a139
  Device-2: AMD Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT]
    vendor: XFX Pine driver: amdgpu v: kernel arch: RDNA-1 pcie: speed: 16 GT/s
    lanes: 16 ports: active: DP-1,HDMI-A-1 off: DP-2 empty: DP-3
    bus-ID: 0a:00.0 chip-ID: 1002:731f
  Device-3: Valve 3D Camera driver: uvcvideo type: USB rev: 3.2
    speed: 5 Gb/s lanes: 1 bus-ID: 2-5.1:4 chip-ID: 28de:2400
  Device-4: Logitech StreamCam
    driver: hid-generic,snd-usb-audio,usbhid,uvcvideo type: USB rev: 3.2
    speed: 5 Gb/s lanes: 1 bus-ID: 6-1.2:4 chip-ID: 046d:0893
  Display: wayland server: X.org v: 1.21.1.11 with: Xwayland v: 23.2.4
    compositor: kwin_wayland driver: X: loaded: modesetting unloaded: fbdev,vesa
    dri: radeonsi gpu: amdgpu display-ID: :1 screens: 1
  Screen-1: 0 s-res: 4480x1440 s-dpi: 96
  Monitor-1: DP-1 pos: primary,top-left res: 2560x1440 dpi: 108
    diag: 690mm (27.15")
  Monitor-2: HDMI-A-1 pos: bottom-r res: 1920x1080 dpi: 96
    diag: 587mm (23.1")
  API: EGL v: 1.5 platforms: device: 0 drv: radeonsi device: 1 drv: swrast
    surfaceless: drv: radeonsi wayland: drv: radeonsi x11: drv: radeonsi
    inactive: gbm
  API: OpenGL v: 4.6 compat-v: 4.5 vendor: amd mesa v: 23.3.5 glx-v: 1.4
    direct-render: yes renderer: AMD Radeon RX 5700 XT (radeonsi navi10 LLVM
    17.0.6 DRM 3.57 6.7.4-1-default) device-ID: 1002:731f
  API: Vulkan v: 1.3.275 surfaces: xcb,xlib,wayland device: 0
    type: discrete-gpu driver: N/A device-ID: 1002:731f

Addional logs

Logs from terminal:

https://gist.github.com/RiQuY/c1cbfb046d1db9ac25cd46dec6dc6882

Output from coredumpctl gdb /media/Almacen/Programas/xr-overlays/wlx-overlay-s/target/release/wlx-overlay-s:

https://gist.github.com/RiQuY/5257a820d21bf1089fa1069d571ee5f1

The coredump was gathered with debuginfod disabled because the openSUSE was having problems downloading the debuginfod, if it is necesary I can run the command again to check if now it is able to download debuginfod.

Performance implications of high resolution displays

I am experiencing extreme performance degradation when trying to display both of my 1440p displays in VR. I haven't dug into the code of the project extensively yet but my gut feeling is that maybe the resolution being displayed is too high and that the actual resolution of the display in VR should be adjusted based off of the headset resolution and the distance to the display in virtual space.

Feat: Configurable OpenXR bindings

Right now, bindings for OpenXR are hardcoded. Let's make them configurable via a file.

[feature request] clock telling you how long you are in vr

xsoverlay has a clock showing the amount of time you have been in steamvr it would be cool if that was a feature on this thanks ozzy

Crate won't compile due to missing libraries on system.

Problem

After cloning the repo and using the cargo run --release command, I- and I'm sure others- have been met with a panic message, and/or compiler errors due to several missing libraries not found on their system.

What are all the locally installable dependencies to allow for this crate to build? and maybe they should be listed in the README.md to be installed before trying to build/run the crate.

Capture specific window

Hi, as a twitch streamer, I would love to have twitch chat attached to my hand to quickly read what people are saying. As of now, the app can only capture desktop. Would it be possible to implement a feature, where for example, inside watch widget, I click on +, PC shows a screenshare portal, and I can choose app window to capture, then below that app, I get a cog icon to open settings for this, and inside those settings, I can choose docking method of the overlay(attached to head, to left/right controller).

[question] how do I make it start with steamvr on linux?

I have a problem where its not starting with steamvr/alvr and I would like to know how to start it with steamvr
any help with this would be appreciated thanks Ozzy

failed to determine active runtime file path for this environment

Thank you for this project. I would love to give it a try with my System (Manjaro Linux, nVidia GPU, Oculus Quest 3).

I've just installed the program from AUR and trying to run it, I get the following error:

$ wlx-overlay-s
INFO [wlx_overlay_s] Welcome to wlx-overlay-s version 0.3.2!
Error [GENERAL |  | OpenXR-Loader] : RuntimeManifestFile::FindManifestFiles - failed to determine active runtime file path for this environment
Error [GENERAL | xrEnumerateInstanceExtensionProperties | OpenXR-Loader] : RuntimeInterface::LoadRuntimes - unknown error
Error [GENERAL | xrEnumerateInstanceExtensionProperties | OpenXR-Loader] : RuntimeInterface::LoadRuntimes - failed to load a runtime
Error [GENERAL | xrEnumerateInstanceExtensionProperties | OpenXR-Loader] : Failed to find default runtime with RuntimeInterface::LoadRuntime()
Error [GENERAL | xrEnumerateInstanceExtensionProperties | OpenXR-Loader] : Failed querying extension properties
WARN [wlx_overlay_s::backend::openxr] Will not use OpenXR: Failed to enumerate OpenXR extensions.
Unable to read VR Path Registry from /home/sebastian/.config/openvr/openvrpaths.vrpath
WARN [wlx_overlay_s::backend::openvr] Will not use OpenVR: Context init failed
ERROR [wlx_overlay_s] No more backends to try

Any suggestions on how I can get starting to fix that? What is likely to cause this?

Feat: Dock overlays to tracked devices.

We need to come up with an intuitive way to set the parent device a given overlay.

Sticking overlays to devices is already handled in auto_movement.

All we need to do is set the OverlayState's spawn_point, spawn_rotation and relative_to.

Active overlays are stored in OverlayContainer, which is normally only accessible from the backend's main loop. However, overlay states can be manipulated from anywhere where AppState is available, by app.tasks.enqueue'ing a TaskType::Overlay. Example here.

Vulkan error creating Vulkan device: ERROR_EXTENSION_NOT_PRESENT

I am sure I am doing something wrong but I can't figure it out.
I tried the appimage as well as building from source.

RUST_BACKTRACE=full ./wlx-overlay-s
Logging to: /tmp/wlx.log
INFO [wlx_overlay_s] Welcome to wlx-overlay-s version 0.3.2!
INFO [wlx_overlay_s] It is Sun Apr 14 13:55:28 2024.
LOG in xrCreateInstance: Instance created
createInfo->applicationInfo.applicationName: wlx-overlay-s
createInfo->applicationInfo.applicationVersion: 0
createInfo->applicationInfo.engineName: wlx-overlay-s
createInfo->applicationInfo.engineVersion: 0
appinfo.detected.engine.name: (null)
appinfo.detected.engine.version: 0.0.0
quirks.disable_vulkan_format_depth_stencil: false
quirks.no_validation_error_in_create_ref_space: true
LOG in xrCreateInstance: Selected devices
Head: 'Nreal Air Glasses'
Eyes: ''
Left: ''
Right: ''
Gamepad: ''
Hand-Tracking Left: ''
Hand-Tracking Right: ''
INFO [wlx_overlay_s::backend::openxr::helpers] Using OpenXR runtime: Monado(XRT) by Collabora et al 'v21.0.0-4304-g084dae3c' 21.0.0
INFO [wlx_overlay_s::backend::openxr] Using environment blend mode: OPAQUE
LOG in xrCreateVulkanInstanceKHR: Creation of VkInstance:
result: VK_SUCCESS
vulkanInstance: 0x55a47b798f30
extensions:
VK_KHR_external_fence_capabilities
VK_KHR_external_memory_capabilities
VK_KHR_external_semaphore_capabilities
VK_KHR_get_physical_device_properties2
VK_EXT_debug_utils
INFO [wlx_overlay_s::graphics] Using vkPhysicalDevice: NVIDIA GeForce RTX 2060 SUPER
LOG in xrGetVulkanGraphicsDeviceKHR: Creation of VkDevice:
result: VK_ERROR_EXTENSION_NOT_PRESENT
vulkanDevice: 0x0
vulkanInstance: 0x55a47b798f30
external_fence_fd: true
external_semaphore_fd: true
timelineSemaphore: true
extensions:
VK_KHR_dedicated_allocation
VK_KHR_external_fence
VK_KHR_external_memory
VK_KHR_external_semaphore
VK_KHR_get_memory_requirements2
VK_KHR_external_memory_fd
VK_KHR_swapchain
VK_EXT_external_memory_dma_buf
VK_EXT_image_drm_format_modifier
VK_KHR_external_semaphore_fd
VK_KHR_external_fence_fd
VK_KHR_image_format_list
VK_KHR_timeline_semaphore
thread 'main' panicked at src/graphics.rs:301:18:
Vulkan error creating Vulkan device: ERROR_EXTENSION_NOT_PRESENT
stack backtrace:
0: 0x55a47a41dda6 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h410d4c66be4e37f9
1: 0x55a47a44b3b0 - core::fmt::write::he40921d4802ce2ac
2: 0x55a47a41ac1f - std::io::Write::write_fmt::h5de5a4e7037c9b20
3: 0x55a47a41db84 - std::sys_common::backtrace::print::h11c067a88e3bdb22
4: 0x55a47a41f257 - std::panicking::default_hook::{{closure}}::h8c832ecb03fde8ea
5: 0x55a47a41efb9 - std::panicking::default_hook::h1633e272b4150cf3
6: 0x55a47a41f6e8 - std::panicking::rust_panic_with_hook::hb164d19c0c1e71d4
7: 0x55a47a41f5c2 - std::panicking::begin_panic_handler::{{closure}}::h0369088c533c20e9
8: 0x55a47a41e2a6 - std::sys_common::backtrace::__rust_end_short_backtrace::hc11d910daf35ac2e
9: 0x55a47a41f314 - rust_begin_unwind
10: 0x55a479ec76b5 - core::panicking::panic_fmt::ha6effc2775a0749c
11: 0x55a479ec7c03 - core::result::unwrap_failed::ha188096f98826595
12: 0x55a479f85fde - wlx_overlay_s::graphics::WlxGraphics::new_openxr::ha73114b6348ea252
13: 0x55a479f3351e - wlx_overlay_s::backend::openxr::openxr_run::h4c3d414342664b27
14: 0x55a479f0fea9 - wlx_overlay_s::main::h2af73d2c044a2635
15: 0x55a479f9d3f3 - std::sys_common::backtrace::__rust_begin_short_backtrace::h91d6d1877081b7c7
16: 0x55a479f121a9 - std::rt::lang_start::{{closure}}::ha277efd1681ca73e
17: 0x55a47a412411 - std::rt::lang_start_internal::h4d236095b69a230b
18: 0x55a479f11ef5 - main
19: 0x7fa8c9ed824a -
20: 0x7fa8c9ed8305 - __libc_start_main
21: 0x55a479ec7f01 - _start
22: 0x0 -

Losing ability to use mouse

First off, amazing project! But I did want to point out an issue I'm experiencing. Randomly I lose the ability to use a controller as a mouse on the projected display. This seems to happen after I finish using other VR applications (games) and then closing them out, but I haven't found a good way to reproduce the issue.

I'm using a quest 2 via ALVR and steamVR on nobara linux, KDE (wayland).