Learn how to use this package by watching our on-demand webinar: Pinpoint, 250 fps, ROS 2 Localization with vSLAM on Jetson

This repository provides a high-performance, best-in-class ROS 2 package for VSLAM (visual simultaneous localization and mapping). This package uses a stereo camera with an IMU to estimate odometry as an input to navigation. It is GPU accelerated to provide real-time, low-latency results in a robotics application. VSLAM provides an additional odometry source for mobile robots (ground based) and can be the primary odometry source for drones.

VSLAM provides a method for visually estimating the position of a robot relative to its start position, known as VO (visual odometry). This is particularly useful in environments where GPS is not available (such as indoors) or intermittent (such as urban locations with structures blocking line of sight to GPS satellites). This method is designed to use left and right stereo camera frames and an IMU (inertial measurement unit) as input. It uses input stereo image pairs to find matching key points in the left and right images; using the baseline between the left and right camera, it can estimate the distance to the key point. Using two consecutive input stereo image pairs, VSLAM can track the 3D motion of key points between the two consecutive images to estimate the 3D motion of the camera--which is then used to compute odometry as an output to navigation. Compared to the classic approach to VSLAM, this method uses GPU acceleration to find and match more key points in real-time, with fine tuning to minimize overall reprojection error.

Key points depend on distinctive features in the left and right camera image that can be repeatedly detected with changes in size, orientation, perspective, lighting, and image noise. In some instances, the number of key points may be limited or entirely absent; for example, if the camera field of view is only looking at a large solid colored wall, no key points may be detected. If there are insufficient key points, this module uses motion sensed with the IMU to provide a sensor for motion, which, when measured, can provide an estimate for odometry. This method, known as VIO (visual-inertial odometry), improves estimation performance when there is a lack of distinctive features in the scene to track motion visually.

SLAM (simultaneous localization and mapping) is built on top of VIO, creating a map of key points that can be used to determine if an area is previously seen. When VSLAM determines that an area is previously seen, it reduces uncertainty in the map estimate, which is known as loop closure. VSLAM uses a statistical approach to loop closure that is more compute efficient to provide a real time solution, improving convergence in loop closure.

There are multiple methods for estimating odometry as an input to navigation. None of these methods are perfect; each has limitations because of systematic flaws in the sensor providing measured observations, such as missing LIDAR returns absorbed by black surfaces, inaccurate wheel ticks when the wheel slips on the ground, or a lack of distinctive features in a scene limiting key points in a camera image. A practical approach to tracking odometry is to use multiple sensors with diverse methods so that systemic issues with one method can be compensated for by another method. With three separate estimates of odometry, failures in a single method can be detected, allowing for fusion of the multiple methods into a single higher quality result. VSLAM provides a vision- and IMU-based solution to estimating odometry that is different from the common practice of using LIDAR and wheel odometry. VSLAM can even be used to improve diversity, with multiple stereo cameras positioned in different directions to provide multiple, concurrent visual estimates of odometry.

To learn more about VSLAM, refer to the Elbrus SLAM documentation.

VSLAM is a best-in-class package with the lowest translation and rotational error as measured on KITTI Visual Odometry / SLAM Evaluation 2012 for real-time applications.

In addition to results from standard benchmarks, we test loop closure for VSLAM on sequences of over 1000 meters, with coverage for indoor and outdoor scenes.

The following table summarizes the per-platform performance statistics of sample graphs that use this package, with links included to the full benchmark output. These benchmark configurations are taken from the Isaac ROS Benchmark collection, based on the ros2_benchmark framework.

For commercial support and access to source code, please contact NVIDIA.

• Isaac ROS Visual SLAM
  • Webinar Available
  • Overview
  • Accuracy
  • Performance
  • Commercial Support & Source Access
  • Table of Contents
  • Latest Update
  • Supported Platforms
    • Docker
  • Coordinate Frames
  • Quickstart
  • Next Steps
    • Try More Examples
    • Customize your Dev Environment
  • Package Reference
    • isaac_ros_visual_slam
      • Usage
      • ROS Parameters
      • ROS Topics Subscribed
      • ROS Topics Published
      • ROS Services Advertised
      • ROS Actions Advertised
  • Troubleshooting
    • Isaac ROS Troubleshooting
    • Troubleshooting Suggestions
  • Updates

Update 2022-03-23: Update Elbrus library and performance improvements

This package is designed and tested to be compatible with ROS 2 Humble running on Jetson or an x86_64 system with an NVIDIA GPU.

Note: Versions of ROS 2 earlier than Humble are not supported. This package depends on specific ROS 2 implementation features that were only introduced beginning with the Humble release.

To simplify development, we strongly recommend leveraging the Isaac ROS Dev Docker images by following these steps. This will streamline your development environment setup with the correct versions of dependencies on both Jetson and x86_64 platforms.

Note: All Isaac ROS Quickstarts, tutorials, and examples have been designed with the Isaac ROS Docker images as a prerequisite.

This section describes the coordinate frames that are involved in the VisualSlamNode. The frames discussed below are oriented as follows:

1. input_base_frame: The name of the frame used to calculate transformation between baselink and left camera. The default value is empty (''), which means the value of base_frame_ will be used. If input_base_frame_ and base_frame_ are both empty, the left camera is assumed to be in the robot's center.
2. input_left_camera_frame: The frame associated with left eye of the stereo camera. Note that this is not the same as the optical frame. The default value is empty (''), which means the left camera is in the robot's center and left_pose_right will be calculated from the CameraInfo message.
3. input_right_camera_frame: The frame associated with the right eye of the stereo camera. Note that this is not the same as the optical frame. The default value is empty (''), which means left and right cameras have identity rotation and are horizontally aligned, so left_pose_right will be calculated from CameraInfo.
4. input_imu_frame: The frame associated with the IMU sensor (if available). It is used to calculate left_pose_imu.

1. Set up your development environment by following the instructions here.

   Note: ${ISAAC_ROS_WS} is defined to point to either /ssd/workspaces/isaac_ros-dev/ or ~/workspaces/isaac_ros-dev/.

2. Clone this repository and its dependencies under ${ISAAC_ROS_WS}/src.

   cd ${ISAAC_ROS_WS}/src
   git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_visual_slam
   git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common
   git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nitros

3. [Terminal 1] Launch the Docker container

   cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
     ./scripts/run_dev.sh ${ISAAC_ROS_WS}

4. [Terminal 1] Inside the container, build and source the workspace:

   cd /workspaces/isaac_ros-dev && \
     colcon build --symlink-install && \
     source install/setup.bash

   (Optional) Run tests to verify complete and correct installation:

   colcon test --executor sequential

   Run the following launch files in the current terminal (Terminal 1):

   ros2 launch isaac_ros_visual_slam isaac_ros_visual_slam.launch.py

5. [Terminal 2] Attach another terminal to the running container for Rviz2

   Attach another terminal to the running container for RViz2.

   cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
     ./scripts/run_dev.sh ${ISAAC_ROS_WS}

   From this second terminal, run Rviz2 to display the output:

   source /workspaces/isaac_ros-dev/install/setup.bash && \
     rviz2 -d src/isaac_ros_visual_slam/isaac_ros_visual_slam/rviz/default.cfg.rviz 

   If you are SSH-ing in from a remote machine, the Rviz2 window should be forwarded to your remote machine.

6. [Terminal 3] Attach the 3rd terminal to start the rosbag

   cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
     ./scripts/run_dev.sh ${ISAAC_ROS_WS}

   Run the rosbag file to start the demo.

   source /workspaces/isaac_ros-dev/install/setup.bash && \
     ros2 bag play src/isaac_ros_visual_slam/isaac_ros_visual_slam/test/test_cases/rosbags/small_pol_test/

7. Result:

   Rviz should start displaying the point clouds and poses like below:

   

Note: By default, IMU data is ignored. To use IMU data in Isaac ROS Visual Slam, please set enable_imu param in your launch file to True. See example below.

To continue your exploration, check out the following suggested examples:

• Tutorial with Isaac Sim
• Tutorial for Visual SLAM using a RealSense camera with integrated IMU

To customize your development environment, reference this guide.

ros2 launch isaac_ros_visual_slam isaac_ros_visual_slam.launch.py

For solutions to problems with Isaac ROS, please check here.

1. If the target frames in rviz are being updated at a lower rate than the input image rate:

   1. Check if the input image frame rate is equal to the value set in the launch file by opening a new terminal and running the command

      ros2 topic hz --window 100 <image_topic_name>

2. If RViz is not showing the poses, check the Fixed Frame value.

3. If you are seeing Tracker is lost. messages frequently, it could be caused by the following issues:

   1. Fast motion causing the motion blur in the frames
   2. Low-lighting conditions.
   3. The wrong camerainfo is being published.

4. For better performance:

   1. Increase the capture framerate from the camera to yield a better tracking result.
   2. If input images are noisy, you can use the denoise_input_images flag in the node.

5. For RealSense cameras-

   1. The firmware of the RealSense camera is 5.14.0 or newer. Check the RealSense firmware-update-tool page for details on how to check the version and update the camera firmware.
   2. You are using the 4.51.1 tagged release for the realsense-ros package.
   3. You are using the parameters set in the RealSense tutorial launch file
   4. If the pose estimate frames in RViz is drifting from actual real-world poses and you see white dots on nearby objects(refer to the screenshot below), this means that the emitter of the RealSense camera is on and it needs to be turned off.
   5. Left and right images being published are not compressed or encoded. Isaac ROS Visual Slam expects raw images.