The Holoscan SDK is part of NVIDIA Holoscan, the AI sensor processing platform that combines hardware systems for low-latency sensor and network connectivity, optimized libraries for data processing and AI, and core microservices to run streaming, imaging, and other applications, from embedded to edge to cloud. It can be used to build streaming AI pipelines for a variety of domains, including Medical Devices, High Performance Computing at the Edge, Industrial Inspection and more.

In previous releases, the prefix Clara was used to define Holoscan as a platform designed initially for medical devices. As Holoscan has grown, its potential to serve other areas has become apparent. With version 0.4.0, we're proud to announce that the Holoscan SDK is now officially built to be domain-agnostic and can be used to build sensor AI applications in multiple domains. Note that some of the content of the SDK (sample applications) or the documentation might still appear to be healthcare-specific pending additional updates. Going forward, domain specific content will be hosted on the HoloHub repository.

Visit the NGC demo website for a live demonstration of some of Holoscan capabilities.

• Documentation
• Using the released SDK
• Building the SDK from source
  • Prerequisites
    • For Clara AGX and NVIDIA IGX Orin Developer Kits (aarch64)
    • For x86_64 systems
  • Recommended: using the run script
  • Cross-compilation
  • Advanced: Docker + CMake
  • Advanced: Local environment + CMake
• Utilities
  • Code coverage
  • Linting
• Troubleshooting
• Repository structure

• The latest SDK user guide is available at https://docs.nvidia.com/clara-holoscan.
• For a full list of Holoscan documentation, visit the Holoscan developer page.

The Holoscan SDK is available as part of the following packages:

• 🐋 The Holoscan container image on NGC includes the Holoscan libraries, GXF extensions, headers, example source code, and sample datasets, as well as all the dependencies that were tested with Holoscan. It is the recommended way to run the Holoscan examples, while still allowing you to create your own C++ and Python Holoscan application.
• 🐍 The Holoscan python wheels on PyPI are the simplest way for Python developers to get started with the SDK using pip install holoscan. The wheels include the SDK libraries, not the example applications or the sample datasets.
• 📦️ The Holoscan Debian package on NGC includes the libraries, headers, example applications and CMake configurations needed for both C++ and Python developers. It does not include sample datasets.

Disclaimer: we only recommend building the SDK from source if you are a developer of the SDK, or need to build the SDK with debug symbols or other options not used as part of the released packages. If you want to modify some code to fit your use case, contribute to HoloHub if it's an operator or application, or file a feature or bug request. If that's not the case, prefer using generated packages.

The Holoscan SDK currently supports the Holoscan Developer Kits (aarch64) as well as x86_64 systems.

Set up your developer kit:

• NVIDIA Clara AGX
• NVIDIA IGX Orin [ES]
• NVIDIA IGX Orin: coming soon

Make sure you have joined the Holoscan SDK Program and, if needed, the Rivermax SDK Program before using the NVIDIA SDK Manager.

• SDK Manager will install Holopack 1.2 as well as the nvgpuswitch.py script. Once configured for dGPU mode, your developer kit will include the necessary components to build the SDK
• For Rivermax support (optional/local development only at this time), GPUDirect drivers need to be loaded manually at this time, see the User Guide.
• Refer to the user guide for additional steps needed to support third-party technologies, such as AJA cards.
• Additional dependencies are required when developing locally instead of using a containerized environment, see details in the section below.

You'll need the following to build applications from source on x86_64:

• OS: Ubuntu 20.04
• NVIDIA GPU
  • Ampere or above recommended for best performance
  • Quadro/NVIDIA RTX necessary for GPUDirect RDMA support
  • Tested with NVIDIA RTX 6000 and NVIDIA RTX A6000
• NVIDIA dGPU drivers: 510.73.08 or above
• For containerized development (recommended):
  • NVIDIA Container Toolkit
• For local development (advanced):
  • See details in the section below.
• For Rivermax support (optional/local development only at this time):
  • NVIDIA ConnectX SmartNIC
  • OFED Network Drivers: 5.8
  • GPUDirect Drivers: 1.1
  • Rivermax SDK: 1.20

Call ./run build within the repository to build the development container and the CMake project.

• If you encounter errors during the CMake build, you can execute ./run clear_cache to remove cache/build/install folders
• Execute ./run build --help for more information
• Execute ./run build --dryrun to see the commands that will be executed
• That command can be broken-up in more granular commands also:

  ./run setup # setup Docker
  ./run build_image # create the development Docker container
  ./run install_gxf # install the GXF package (default location is `.cache/gxf`)
  ./run build # run the CMake configuration, build, and install steps

Call the ./run launch command to start and enter the development container.

• You can run from the install or build tree by passing the working directory as an argument (ex: ./run launch install)
• Execute ./run launch --help for more information
• Execute ./run launch --dryrun to see the commands that will be executed

Run the examples inside the container by running their respective commands listed within each directory README file.

While the development Dockerfile does not currently support true cross-compilation, you can compile the Holoscan SDK for the developer kits (arm64) from a x86_64 host using an emulation environment.

1. Install qemu
2. Clear your build cache: ./run clear_cache
3. Rebuild for linux/arm64 using --platform or HOLOSCAN_BUILD_PLATFORM:
   • ./run build --platform linux/arm64
   • HOLOSCAN_BUILD_PLATFORM=linux/arm64 ./run build

You can then copy the install folder generated by CMake to a developer kit with a configured environment or within a container to use for running and developing applications.

The run script mentioned above is helpful to understand how Docker and CMake are configured and run, as commands will be printed when running it or using --dryrun. We recommend looking at those commands if you want to use Docker and CMake manually, and reading the comments inside the script for details about each parameter (specifically the build() and launch() methods).

Disclaimer: the risk of this section getting out of date is high since this is not actively maintained and tested at this time. Look at the dependencies pulled in the Dockerfile and the CMake commands in the run script if you run into issues.

To build on your local environment, you'll need to install the dependencies below in addition to the ones listed in the prerequisites:

• CUDA Toolkit: 11.6.1
  • arm64: already installed by SDK Manager on the Developer Kits
  • x86_64: follow official instructions
• TensorRT 8.2.3
  • arm64: already installed by SDK Manager on the Developer Kits
  • x86_64: follow official instructions
• python-dev: 3.8.10
  • sudo apt install python3.8-dev
• apt dependencies (CMake, patchelf, OpenGL, Xorg stack...)
  • see the base stage in Dockerfile
  • see the dev stage in Dockerfile
• ONNX Runtime: 1.12.1
  • download: see the onnxruntime-downloader stage in Dockerfile
  • install: see the dev stage in Dockerfile
• Vulkan SDK: 1.3.216
  • download & build: see the vulkansdk-builder stage in Dockerfile
  • install: see the dev stage in Dockerfile
• GXF: 2.5
  • call ./run install_gxf or look at its content

You'll then need CMake to find those dependencies during configuration. Use CMAKE_PREFIX_PATH, CMAKE_LIBRARY_PATH, and/or CMAKE_INCLUDE_PATH if they're not installed in default system paths and cannot automatically be found.

# Configure
cmake -S $source_dir -B $build_dir \
  -G Ninja \
  -D CMAKE_BUILD_TYPE=Release \
  -D CUDAToolkit_ROOT:PATH="/usr/local/cuda"

# Build
cmake --build $build_dir -j

# Install
cmake --install $build_dir --prefix $install_dir

The commands to run the examples are then the same as in the dockerized environment, and can be found in the respective source directory READMEs.

Some utilities are available in the scripts folder, others closer to the built process are listed below:

To generate a code coverage report use the following command after completing setup:

./run coverage

Open the file build/coverage/index.html to access the interactive coverage web tool.

Run the following command to run various linting tools on the repository:

./run lint # optional: specify directories

Note: Run run lint --help to see the list of tools that are used. If a lint command fails due to a missing module or executable on your system, you can install it using python3 -m pip install <tool>.

Visual Studio Code can be utilized to develop Holoscan SDK. The .devcontainer folder holds the configuration for setting up a development container with all necessary tools and libraries installed.

The ./run script contains vscode and vscode_remote commands for launching Visual Studio Code in a container or from a remote machine, respectively.

• To launch Visual Studio Code in a dev container, use ./run vscode.
• To attach to an existing dev container from a remote machine, use ./run vscode_remote. For more information, refer to the instructions from ./run vscode_remote -h.

Once Visual Studio Code is launched, the development container will be built and the recommended extensions will be installed automatically, along with CMake being configured.

For manual configuration of CMake, open the command palette (Ctrl + Shift + P) and run the CMake: Configure command.

The source code in the development container can be built by either pressing Ctrl + Shift + B or executing Tasks: Run Build Task from the command palette (Ctrl + Shift + P).

To debug the source code in the development container, open the Run and Debug view (Ctrl + Shift + D), select a debug configuration from the dropdown list, and press F5 to initiate debugging.

Enable permissions to your X server from Docker, either:

• Passing -u $(id -u):$(id -g) to docker run, or
• Running xhost +local:docker on your host

You may encounter the error message if the Holoscan Application runs on a Virtual Machine (by a Cloud Service Provider) or without a physical display attached. If you want to run OpenGL applications that use GPU on x11 (e.g., VNC or NoMachine), the following environment variables need to be set before executing the application to offload the rendering to GPU.

export __NV_PRIME_RENDER_OFFLOAD=1
export __GLX_VENDOR_LIBRARY_NAME=nvidia

Ensure all your application connections in the yaml file (nvidia::gxf::Connection) refer to entities or components defined within. This can occur when attempting to remove a component and not cleaning up the stale connections.

Ensure your entity or component is not an orphan, but is connected to a nvidia::gxf::Connection.

These errors indicate that you don't have AJA support in your environment.

2022-06-09 18:45:13.826 ERROR gxf_extensions/aja/aja_source.cpp@80: Device 0 not found.
2022-06-09 18:45:13.826 ERROR gxf_extensions/aja/aja_source.cpp@251: Failed to open device 0

Double check that you have installed the AJA ntv2 driver, loaded the driver after every reboot, and that you have specified --device /dev/ajantv20:/dev/ajantv20 in the docker run command if you’re running a docker container.

These errors may indicate that you need to reconfigure your format converter's num_block number.

2022-06-09 18:47:30.769 ERROR gxf_extensions/format_converter/format_converter.cpp@521: Failed to allocate memory for the channel conversion
2022-06-09 18:47:30.769 ERROR gxf_extensions/format_converter/format_converter.cpp@359: Failed to convert tensor format (conversion type:6)

Try increasing the current num_block number by 1 in the yaml file for all format converter entities. This may happen if your yaml file was configured for running with RDMA and you have decided to disable RDMA.

Some of those errors may occur when running the V4L2 codelet:

Failed to open device, OPEN: No such file or directory

Ensure you have a video device connected (ex: USB webcam) and listed when running ls -l /dev/video*.

Failed to open device, OPEN: Permission denied

This means the /dev/video* device is not available to the user from within docker. You can make it publicly available with sudo chmod 666 /dev/video* from your host.

The error can happen when running in a multi-GPU environment:

[2023-02-10 10:42:31.039] [holoscan] [error] [gxf_wrapper.cpp:68] Exception occurred for operator: 'holoviz' - Cuda driver error 101 (CUDA_ERROR_INVALID_DEVICE): invalid device ordinal
2023-02-10 10:42:31.039 ERROR gxf/std/entity_executor.cpp@509: Failed to tick codelet holoviz in entity: holoviz code: GXF_FAILURE

This is due to the fact that operators in your application run on different GPUs, depending on their internal implementation, preventing data from being freely exchanged between them.

To fix the issue, either:

1. Configure or modify your operators to copy data across the appropriate GPUs (using cuPointerGetAttribute and cuMemCpyPeerAsync() internally). This comes at a cost and should only be done when leveraging multiple GPUs is required for improving performance (parallel processing).
2. Configure or modify your operators to use the same GPU (using cudaSetDevice() internally).
3. Restrict your environment using CUDA_VISIBLE_DEVICES to only expose a single GPU:

   export CUDA_VISIBLE_DEVICES=<GPU_ID>

Note that this generally occurs because the HolovizOp operator needs to use the GPU connected to the display, but other operators in the Holoscan pipeline might default to another GPU depending on their internal implementation. The index of the GPU connected to the display can be found by running nvidia-smi from a command prompt and looking for the GPU where the Disp.A value shows On. In the example below, the GPU 0 should be used before passing data to HolovizOp.

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA RTX A5000    On   | 00000000:09:00.0  On |                  Off |
| 30%   29C    P8    11W / 230W |    236MiB / 24564MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA RTX A5000    On   | 00000000:10:00.0 Off |                  Off |
| 30%   29C    P8    11W / 230W |    236MiB / 24564MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

Note: Holoviz should support copying data across CUDA devices in a future release.

The repository is organized as such:s

• cmake/: CMake configuration files
• data/: directory where data will be downloaded
• examples/: source code for the examples
• gxf_extensions/: source code for the holoscan SDK gxf codelets
• include/: source code for the holoscan SDK core
• modules/: source code for the holoscan SDK modules
• patches/: patch files applied to dependencies
• python/: python bindings for the holoscan SDK
• scripts/: utility scripts
• src/: source code for the holoscan SDK core
• tests/: tests for the holoscan SDK