Summary: Our method, DVMPC can realize the navigation with obstacle avoidance by only using an RGB image. Our control policy, PoliNet is trained under the same objectives as Model Predictive Control(MPC), which includes, image loss, traversability loss and reference loss.

Please see the website (http://svl.stanford.edu/projects/dvmpc/) for more technical details. This repository is intended for distribution of the code and its instruction.

"Deep Visual MPC-Policy Learning for Navigation"

Ubuntu 16.04

Chainer 4.1.0

Python Pillow 1.1.7

ROS KINETIC(http://wiki.ros.org/kinetic)

Nvidia GPU

We are providing DVMPC, which can realize the navigation with obstacle avoidance by only using an RGB image.

git clone https://github.com/NHirose/DVMPC.git

DVMPC can only accept the 360-degree camera image to capture the environment in front of the robot. We highly recommend to use RICOH THETA S.(https://theta360.com/en/about/theta/s.html) Please put the camera in front of your device(robot) at the height 0.460 m not to caputure your device itself and connect with your PC by USB cable.

To turn on RICOH THETA S as the live streaming mode, please hold the bottom buttom at side for about 5 senconds and push the top buttom.(Detail is shown in the instrunction sheet of RICOH THETA S.)

To capture the image from RICOH THETA S, we used the open source in ROS, cv_camera_node(http://wiki.ros.org/cv_camera). The subscribed topic name of the image is "/cv_camera_node1/image". We recommend that the flame rate of image is 3 fps.

DVMPC can follow the visual trajectory, which is consructed by the time consecutive 360-degree images. So, before the navigation, you need to collect the visual trajectory by tele-operation of the robot. Our code subscribes the visual trajectory as "/cv_camera_node2/image_ref". Therefore, you need to feed the subgoal image from the visual trajectory into "/cv_camera_node2/image_ref".

The last process to have the navigation is just to run our algorithm.

python dvmpc.py

The published topic name for the velocity reference is "/cmd_vel_mux/input/ref_GONetpp". "img_ref" is the topic name for the current and subgoal images. And, front and back predicted images for 8 steps by VUNet-360 are published as "img_genf" and "img_genb". If your implementation sounds correct, the 8-th predicted images should be similar to the subgoal image.

The codes provided on this page are published under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License(https://creativecommons.org/licenses/by-nc-sa/3.0/). This means that you must attribute the work in the manner specified by the authors, you may not use this work for commercial purposes and if you alter, transform, or build upon this work, you may distribute the resulting work only under the same license. If you are interested in commercial usage you can contact us for further options.

If you use DVMPC's software or database, please cite:

@article{hirose2019deep,
title={Deep visual mpc-policy learning for navigation},
author={Hirose, Noriaki and Xia, Fei and Mart{'\i}n-Mart{'\i}n, Roberto and Sadeghian, Amir and Savarese, Silvio},
journal={IEEE Robotics and Automation Letters},
volume={4},
number={4},
pages={3184--3191},
year={2019},
publisher={IEEE}
}