A gaze enhanced perception-aware assistive teleoperation framework.

Video Links: youtube or bilibili

GPA-Teleoperation is a gaze enhanced perception-aware assistive teleoperation framework. It simultaneously ensures safety and increases the perception awareness to environment of interest.

Authors: Qianhao Wang ([email protected]), Botao He ([email protected]) and Fei Gao from the ZJU Fast Lab.

Paper: GPA-Teleoperation: Gaze Enhanced Perception-aware Safe Assistive Aerial Teleoperation, Qianhao Wang*, Botao He*, Zhiren Xun and Fei Gao, Published in IEEE Robotics and Automation Letters (RAL).

git clone https://github.com/ZJU-FAST-Lab/GPA-Teleoperation.git
cd GPA-Teleoperation

We use CUDA to render depth. Please remember to also change the 'arch' and 'code' flags in the line of

  set(CUDA_NVCC_FLAGS 
      # set this according to your cuda version
      -gencode=arch=compute_86,code=sm_86 ;
  ) 

in CMakeList.txt in the package local_sensing. If you encounter compiling error due to different Nvidia graphics card you use or you can not see proper depth images as expected, you can check the right code via link1 or link2.

The Python package tkinter is used to create the videography interface. To install tkinter,

pip install pynput

catkin_make
source devel/setup.sh

roslaunch gpa_teleop sim.launch

# open a new terminal under the same directory
python3 src/gaze_sim/gaze_sim.py

Our GUI provides gaze simulation using the mouse. The gif below shows a demo that using mouse to indicate the human intention.

Note: Due to the display updates in some versions of OpenCV 4, you may need to firstly move the mouse out of the display region and then scroll the wheel to adjust the speed

We use MINCO as our trajectory representation.

We use EGO-Planner-v2 as the target drone planner.

@ARTICLE{9720920,
  author={Wang, Qianhao and He, Botao and Xun, Zhiren and Xu, Chao and Gao, Fei},
  journal={IEEE Robotics and Automation Letters}, 
  title={GPA-Teleoperation: Gaze Enhanced Perception-Aware Safe Assistive Aerial Teleoperation}, 
  year={2022},
  volume={7},
  number={2},
  pages={5631-5638},
  doi={10.1109/LRA.2022.3153898}}