yshui / index_camera_passthrough Goto Github PK

View Code? Open in Web Editor NEW

46.0 3.0 1.0 536 KB

Experimental Valve Index camera passthrough for Linux

License: MIT License

Rust 97.23% GLSL 1.82% Nix 0.96%

openvr steamvr vr linux passthrough

index_camera_passthrough's Introduction

Index camera passthrough

Warning: This is still a work in progress, you could get motion sickness if you try it now

The problem that the Index camera doesn't work on Linux has been there for a long time, see ValveSoftware/SteamVR-for-Linux#231. And Valve doesn't seem to be willing to address it. So I decided to throw something together.

Features

Stereo overlay: the overlay in your game world that acts as a portal to real world. Meaning you see in 3D. (disabled by default, see the example config file for how to enable and more options.)
You can configure the overlay to be in one place, or stay in front of you.
Use camera calibration data from your Steam installation.
Show/hide passthrough with button presses

TODO

Add option to make overlay follow controller.
(Unrealistic) implement Valve's "3D" passthrough. To do this we essentially need to do 3D reconstruction from the stereo camera. There are existing methods, but will be really challenging to implement.

Contribute

You can test this out and report your experience to help this improve.

If you have any suggestions about features, or how to make the passthrough look better, please let me know. I am not a graphics programmer and am trying my best to get things work, but solutions I came up with is definitely not going to be as good as things can be.

Please help me out.

Build instruction

You can install this program from crates.io:

cargo install index_camera_passthrough

To build this program, you need:

Rust (How to install, you need to select the nightly channel)
OpenVR
Vulkan

in the repository first, then run

cargo build --release

Usage

Run from Steam library

After you have built the program, copy it to /usr/local/bin

cp ./target/release/index_camera_passthrough /usr/local/bin

And then add the index_camera_passthrough.desktop file to your Steam Library.

Run directly

To run this program, you can either

cargo run

or run the binary directly

./target/release/index_camera_passthrough

index_camera_passthrough's People

Contributors

Stargazers

Watchers

Forkers

scrumplex

index_camera_passthrough's Issues

Missing v4l2 dev results in nondescript error

If the v4l2 device expected in the config is not found this is the result

Error: No such file or directory (os error 2)

panic when running in release mode

When running in release, it panics after showing a few frames from the camera.

Found matching device: AMD Radeon RX 6800 XT (RADV NAVI21)
thread 'main' panicked at /home/crispypin/.cargo/git/checkouts/vulkano-0df1d54ea6cf1ce4/d828801/vulkano/src/command_buffer/allocator.rs:440:41:
called `Option::unwrap()` on a `None` value
stack backtrace:
   0: rust_begin_unwind
             at /rustc/2831701757eb7b3105eda26a306c2f3a97e2664b/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/2831701757eb7b3105eda26a306c2f3a97e2664b/library/core/src/panicking.rs:72:14
   2: core::panicking::panic
             at /rustc/2831701757eb7b3105eda26a306c2f3a97e2664b/library/core/src/panicking.rs:127:5
   3: vulkano::command_buffer::allocator::Entry::allocate
   4: <vulkano::command_buffer::allocator::StandardCommandBufferAllocator as vulkano::command_buffer::allocator::CommandBufferAllocator>::allocate
   5: vulkano::command_buffer::auto::builder::AutoCommandBufferBuilder<L>::begin
   6: index_camera_passthrough::pipeline::submit_cpu_image
   7: index_camera_passthrough::pipeline::Pipeline::run
   8: index_camera_passthrough::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Requires nightly/pre-release rust

With a standard rust install, building gives the error

error[E0554]: `#![feature]` may not be used on the stable release channel
 --> openvr_sys/src/lib.rs:1:1
  |
1 | #![feature(untagged_unions)]
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

For more information about this error, try `rustc --explain E0554`.

The readme should note the requirement of pre-release rust.

Missing file: splash.png

It seems a file named splash.png should be included in the source, but is not.

error: couldn't read /home/claire/GIT/index_camera_passthrough/splash.png: No such file or directory (os error 2)
  --> src/main.rs:50:30
   |
50 | static SPLASH_IMAGE: &[u8] = include_bytes!(concat!(env!("CARGO_MANIFEST_DIR"), "/splash.png"));
   |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: this error originates in the macro `include_bytes` (in Nightly builds, run with -Z macro-backtrace for more info)

error: could not compile `index_camera_passthrough` due to previous error

Creating a blank 1920×960 png works fine.

Crash with OG Vive camera

Using the OG Vive (single camera) by manually setting the camera path in config causes a crash in either VR backend.

[2024-03-05T15:55:16Z INFO  index_camera_passthrough] width          : 612
    height         : 460
    fourcc         : YUYV
    field          : progressive
    stride         : 1224
    size           : 563040
    colorspace     : sRGB
    quantization   : default
    transfer       : Rec. 709 transfer function
...
[2024-03-05T15:55:16Z WARN  index_camera_passthrough] No camera parameters found
[2024-03-05T15:55:16Z INFO  index_camera_passthrough::pipeline] Adjusted FOV: [[1.19, 1.19], [1.19, 1.19]]
Error: copy_buffer_to_image_info: `regions[0].buffer_offset` plus the number of bytes being copied is greater than `src_buffer.size()` (Vulkan VUIDs: VUID-VkCopyBufferToImageInfo2-pRegions-00171)

Backtrace (backtrace.log):

...
[2024-03-05T15:25:41Z TRACE index_camera_passthrough] state: Capturing
[2024-03-05T15:25:41Z TRACE index_camera_passthrough] getting camera frame
[2024-03-05T15:25:41Z TRACE index_camera_passthrough] got camera frame Instant { tv_sec: 7818, tv_nsec: 405975511 }
[2024-03-05T15:25:41Z TRACE index_camera_passthrough] state: Capturing
[2024-03-05T15:25:41Z TRACE index_camera_passthrough] getting camera frame
Error: copy_buffer_to_image_info: `regions[0].buffer_offset` plus the number of bytes being copied is greater than `src_buffer.size()`

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /home/lea/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.75/src/error.rs:551:25
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/d18480b84fdbf1efc34f62070951334aa833d761/library/core/src/result.rs:1964:27
   2: index_camera_passthrough::pipeline::Pipeline::submit_cpu_image
             at ./src/pipeline.rs:188:9
   3: index_camera_passthrough::pipeline::Pipeline::run
             at ./src/pipeline.rs:323:26
   4: index_camera_passthrough::main
             at ./src/main.rs:415:38
[2024-03-05T15:25:41Z TRACE index_camera_passthrough] got camera frame Instant { tv_sec: 7818, tv_nsec: 422727511 }
[2024-03-05T15:25:41Z TRACE index_camera_passthrough] state: Capturing
[2024-03-05T15:25:41Z TRACE index_camera_passthrough] getting camera frame

Changed config values:

backend = "OpenXR"
camera_device = "/dev/video0"

No camera parameters found

Built from source, throws this when i try to run:
[2024-03-15T01:47:27Z INFO index_camera_passthrough] width : 1920
height : 960
fourcc : YUYV
field : progressive
stride : 3840
size : 3686400
colorspace : sRGB
quantization : default
transfer : Rec. 709 transfer function

[2024-03-15T01:47:27Z INFO index_camera_passthrough] OpenVR
Found matching device: NVIDIA GeForce RTX 4090
[2024-03-15T01:47:28Z WARN index_camera_passthrough] No camera parameters found
[2024-03-15T01:47:28Z INFO index_camera_passthrough::vrapi] Dropping overlay handle
[2024-03-15T01:47:28Z INFO index_camera_passthrough::openvr] Shutdown OpenVR
Error: a non-validation error occurred

Caused by:
0: allocating memory for the image failed
1: finding a suitable memory type failed

Edit:Missed the last two lines.

Program runs without errors but the overlay doesn't appear

I built from a current checkout (cloned right before I built).
I used rust nightly to compile.

I'm using Arch Linux, I did have some packages out of date, but not ones relating to steam or graphics.
I have a NVIDIA GTX 1070 using proprietary drivers and I'm running steam with the native runtime.

I tried the menu button binding and the trigger binding. I have camera disabled in the SteamVR settings.
Steam and SteamVR were up to date. SteamVR version is 1.22.6 Beta. I couldn't find a version number for Steam.

I was using the "Stereo" display mode.

error: failed to run custom build command for v4l2-sys-mit v0.2.0

Hello, I'm sorry if the solution to this issue is obvious. I've looked up the error, but I couldn't find anything.

OS: Arch Linux
Rust Nightly
Installed via rustup from the official source
I have openvr, and nvidia-utils installed

I'm getting this error when trying to build index_camera_passthrough

error: failed to run custom build command for v4l2-sys-mit v0.2.0

Caused by:
  process didn't exit successfully: /tmp/cargo-installSET54P/release/build/v4l2-sys-mit-057ded0b20d221c6/build-script-build (exit status: 101)
  --- stderr
  thread 'main' panicked at /home/matrix/.cargo/registry/src/index.crates.io-6f17d22bba15001f/proc-macro2-1.0.69/src/fallback.rs:817:9:
  "v4l2_pix_format_union_(anonymous_at_/usr/include/linux/videodev2_h_500_2)" is not a valid Ident
  note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: failed to compile index_camera_passthrough v0.1.0, intermediate artifacts can be found at /tmp/cargo-installSET54P.

Use more appropriate vulkano heap sizes

Hi, I was mostly doing my own research on this and I thought you might be interested.

Vulkano's default StandardMemoryAllocator allocates around 576MB of memory even if you don't upload anything. This is not great for an overlay app that just wants to display a 720p image.

A typical AMD card is known to expose 3 heaps; 1 DEVICE_LOCAL, 1 HOST_VISIBLE and usually 1 smaller (<1GB) DEVICE_LOCAL | HOST_VISIBLE.

StandardMemoryAllocator::default_new() will allocate 256MB per large (>1GB) heap and 64MB for the small heap, making it seem like your app is using 576MB memory (+ openxr swapchains, as they are not managed by vulkano).

The answer for me was to explicitly set the heap sizes instead of using new_default().

https://github.com/galister/wlx-overlay-s/blob/6842872314ad444b38fdd15b90b784964c0fbdae/src/graphics.rs#L1172-L1209

Events module not included

The build fails with:

error[E0583]: file not found for module `events`
 --> src/main.rs:4:1
  |
4 | mod events;
  | ^^^^^^^^^^^
  |
  = help: to create the module `events`, create file "src/events.rs" or "src/events/mod.rs"

error: couldn't read /home/chris/Projects/index_camera_passthrough/splash.png: No such file or directory (os error 2)
  --> src/main.rs:50:30
   |
50 | static SPLASH_IMAGE: &[u8] = include_bytes!(concat!(env!("CARGO_MANIFEST_DIR"), "/splash.png"));
   |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: this error originates in the macro `include_bytes` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0433]: failed to resolve: could not find `Action` in `events`
   --> src/main.rs:522:21
    |
522 |             events::Action::ShowOverlay => vroverlay
    |                     ^^^^^^ could not find `Action` in `events`

error[E0433]: failed to resolve: could not find `Action` in `events`
   --> src/main.rs:526:21
    |
526 |             events::Action::HideOverlay => vroverlay
    |                     ^^^^^^ could not find `Action` in `events`

error[E0433]: failed to resolve: could not find `State` in `events`
   --> src/main.rs:367:29
    |
367 |     let mut state = events::State::new(cfg.toggle_button, cfg.open_delay);
    |                             ^^^^^ not found in `events`
    |
help: consider importing this enum
    |
10  | use serde_json::ser::State;
    |

It seems b868aa5 didn't bring in the required ~~package~~ file

Failure to allocate image on GTX 1080

For some reason the allocation of yuv_texture in pipeline.rs fails:

Error: a non-validation error occurred

Caused by:
    0: allocating memory for the image failed
    1: finding a suitable memory type failed

The Vulkan wrapper isn't finding the right memory type to create the image. I might be overlooking a flag but it seems like a suitable memory type is present though:

$ vulkaninfo
...
memoryHeaps: count = 3
        memoryHeaps[0]:
                size   = 8589934592 (0x200000000) (8.00 GiB)
                budget = 4746379264 (0x11ae80000) (4.42 GiB)
                usage  = 0 (0x00000000) (0.00 B)
                flags: count = 1
                        MEMORY_HEAP_DEVICE_LOCAL_BIT
        memoryHeaps[1]:
                size   = 25176649728 (0x5dca53000) (23.45 GiB)
                budget = 25176649728 (0x5dca53000) (23.45 GiB)
                usage  = 0 (0x00000000) (0.00 B)
                flags:
                        None
        memoryHeaps[2]:
                size   = 257949696 (0x0f600000) (246.00 MiB)
                budget = 198508544 (0x0bd50000) (189.31 MiB)
                usage  = 59441152 (0x038b0000) (56.69 MiB)
                flags: count = 1
                        MEMORY_HEAP_DEVICE_LOCAL_BIT
memoryTypes: count = 11
        # ... more memory types ...
        memoryTypes[8]:
                heapIndex     = 1
                propertyFlags = 0x0006: count = 2
                        MEMORY_PROPERTY_HOST_VISIBLE_BIT
                        MEMORY_PROPERTY_HOST_COHERENT_BIT
                usable for:
                        IMAGE_TILING_OPTIMAL:
                                None
                        IMAGE_TILING_LINEAR:
                                color images
                                (non-sparse, non-transient)
        memoryTypes[9]:
                heapIndex     = 1
                propertyFlags = 0x000e: count = 3
                        MEMORY_PROPERTY_HOST_VISIBLE_BIT
                        MEMORY_PROPERTY_HOST_COHERENT_BIT
                        MEMORY_PROPERTY_HOST_CACHED_BIT
                usable for:
                        IMAGE_TILING_OPTIMAL:
                                None
                        IMAGE_TILING_LINEAR:
                                color images
                                (non-sparse, non-transient)
        memoryTypes[10]:
                heapIndex     = 2
                propertyFlags = 0x0007: count = 3
                        MEMORY_PROPERTY_DEVICE_LOCAL_BIT
                        MEMORY_PROPERTY_HOST_VISIBLE_BIT
                        MEMORY_PROPERTY_HOST_COHERENT_BIT
                usable for:
                        IMAGE_TILING_OPTIMAL:
                                None
                        IMAGE_TILING_LINEAR:
                                color images
                                (non-sparse, non-transient)

The overlay does work if I just change the memory type filter for this one allocation to force the allocation to run:

diff --git a/src/pipeline.rs b/src/pipeline.rs
index c632b4b..b6b17bd 100644
--- a/src/pipeline.rs
+++ b/src/pipeline.rs
@@ -226,8 +226,7 @@ impl Pipeline {
                 ..Default::default()
             },
             AllocationCreateInfo {
-                memory_type_filter: MemoryTypeFilter::HOST_SEQUENTIAL_WRITE
-                    | MemoryTypeFilter::PREFER_DEVICE,
+                memory_type_filter: MemoryTypeFilter::PREFER_DEVICE,
                 ..Default::default()
             },
         )?;

System information for reference:

GPU driver: nvidia 545.29.06
kernel: Linux 6.7.0-arch3-1

Build in release mode can't find vulkan layer, debug works

bones@rog ~/Programs/git/xr-overlays/index_camera_passthrough $ RUST_LOG=debug cargo run --release
    Finished release [optimized] target(s) in 0.12s
     Running `target/release/index_camera_passthrough`
[2024-01-30T07:09:18Z INFO  index_camera_passthrough] width          : 1920
    height         : 960
    fourcc         : YUYV
    field          : progressive
    stride         : 3840
    size           : 3686400
    colorspace     : sRGB
    quantization   : default
    transfer       : Rec. 709 transfer function
    
[2024-01-30T07:09:18Z DEBUG index_camera_passthrough] loading splash
[2024-01-30T07:09:18Z DEBUG index_camera_passthrough] splash loaded
[2024-01-30T07:09:18Z INFO  index_camera_passthrough] OpenXR
LOG in xrCreateInstance: Instance created
        createInfo->applicationInfo.applicationName: Camera
        createInfo->applicationInfo.applicationVersion: 0
        createInfo->applicationInfo.engineName: engine
        createInfo->applicationInfo.engineVersion: 0
        appinfo.detected.engine.name: (null)
        appinfo.detected.engine.version: 0.0.0
        quirks.disable_vulkan_format_depth_stencil: false
        quirks.no_validation_error_in_create_ref_space: true
LOG in xrCreateInstance: Selected devices
        Head: 'Index HMD'
        Eyes: '<none>'
        Left: 'Valve Index Controller'
        Right: 'Valve Index Controller'
        Gamepad: '<none>'
        Hand-Tracking Left: 'Valve Index Controller'
        Hand-Tracking Right: 'Valve Index Controller'
LOG in xrCreateVulkanInstanceKHR: Creation of VkInstance:
        result: VK_ERROR_LAYER_NOT_PRESENT
        vulkanInstance: 0x0
        extensions:
                VK_KHR_external_fence_capabilities
                VK_KHR_external_memory_capabilities
                VK_KHR_external_semaphore_capabilities
                VK_KHR_get_physical_device_properties2
                VK_EXT_debug_utils
                VK_KHR_device_group_creation
                VK_KHR_display
                VK_KHR_get_display_properties2
                VK_KHR_get_surface_capabilities2
                VK_KHR_portability_enumeration
                VK_KHR_surface
                VK_KHR_surface_protected_capabilities
                VK_KHR_wayland_surface
                VK_KHR_xcb_surface
                VK_KHR_xlib_surface
                VK_EXT_acquire_drm_display
                VK_EXT_acquire_xlib_display
                VK_EXT_debug_report
                VK_EXT_direct_mode_display
                VK_EXT_display_surface_counter
                VK_EXT_surface_maintenance1
                VK_EXT_swapchain_colorspace
                VK_LUNARG_direct_driver_loading
Error: vulkan error: Layer specified does not exist

Caused by:
    Layer specified does not exist