Associated Paper: Robust Navigation with Loomo

This repository shows the implementation of a robust and modular Perception-Pipeline. This Perception module was developed to be implemented for real-time on Loomo Segway Robot. Furthermore, this Repository can also serve as a benchmark to test the performance of different perception algorithms on Single Person Tracking Videos/Img Sequences.

This pipeline propose a modular implementation combining Yolov5 and OpenPifPaf for the Detection module. Deepsort, SiamRPN++ and Stark for the Tracking module and Finally Deep Visual Re-Identification with Confidence for the ReID.

• Clone the repository recursively:

   git clone --recurse-submodules
  

  If you already cloned and forgot to use --recurse-submodules you can run git submodule update --init

• Look at the Install Procedure Described at the bottom

• Requirements:

  • Python3
  • OpenCV
  • Pytorch
  • torchvision
  • Openpifpaf

├── Benchmark      # Videos/Img sequences + groundtruth bbox
├── libs     # Folder for the installation of external libraries (mmtracking, deep-person-reid)
├── python         # Python files containing perception module
├── Results        # Folder to store the videos and bbox resulting from inference
├── others         # Other materials / Android app APK
└── README.md

Check the other ReadMe file (add link) to get more info about the perception module and configurations

Stark: Learning Spatio-Temporal Transformer for Visual Tracking

cd src/perceptionloomo/mmtracking
mkdir checkpoints
wget https://download.openmmlab.com/mmtracking/sot/stark/stark_st2_r50_50e_lasot/stark_st2_r50_50e_lasot_20220416_170201-b1484149.pth

SiameseRPN++: Evolution of Siamese Visual Tracking With Very Deep Networks

wget https://download.openmmlab.com/mmtracking/sot/siamese_rpn/siamese_rpn_r50_1x_lasot/siamese_rpn_r50_20e_lasot_20220420_181845-dd0f151e.pth

To download data to the Benchmark folder use the command: wget <link_data> in the Benchmark folder.

• LaSOT: Large scale dataset for Single Image Tracking (SOT) as our detection algorithm is only trained on humans we are only interested on the person category available for download here
• OTB100: Img sequences from various different sources with provided categories for tracking challenges (Illumination variations, Body deformation...). Only some sequences are relevant for out perception module (Human, Skater, BlurBody, Basketball...)
• Loomo Dataset provides 8 + 16 Videos with provided ground truth from real-life experiments recordings. The Dataset is available for download at this link

Launch the run.py script (python/scripts) to try the default perception module configuration

- In case you decide to use the Stark tracker use pythonn -W ignore run.py to avoid printing the warning messages.

(To get more details on how to change the configurations and input file check the ReadMe.md inside python directory)

Make sure to be connected to the same WiFi, get the ip adress of Loomo from settings. Then connect using adb connect <ip Loomo> and check connection is working using adb devices.

Run the AlgoApp on Loomo and press on button Start Algo to allow for socket connection, you should now see the camera on Loomo's screen.

Before trying to launch the app on loomo make sure to have the same downscale parameter on Loomo and in the config file: cfg_perception.yaml. To see the config on loomo use the command: adb shell cat /sdcard/follow.cfg

Easiest way to start is to change the ip-adress of loomo in the config file (python/src/perceptionloomo/configs/cfg_perception.yaml) and launch the script to run the algorithm on loomo python3 python/scripts/loomo.py

First set up a python >= 3.7 virtual environment:

cd <desired-dir>
python3 -m venv <desired venv name>
source <name_venv>/bin/activate

Make sure to install python 3 in case you do not have it.

In the root directory: after creating a virtual environment run the following commands to install pip packages pip install -r requirements.txt

In the root directory do to the libs directory and install the following repositories. (In case you didn't clone with submodule run: git submodule update --init to download submodule for deep-person-reid)

cd libs
cd deep-person-reid/
pip install -r requirements.txt
python setup.py develop

cd ../
git clone [email protected]:open-mmlab/mmtracking.git
cd mmtracking 

Follow the install instructions from the official documentation

pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu102/torch1.10.0/index.html
pip install mmdet
pip install -r requirements/build.txt
pip install -v -e .

In the Perception-Pipeline/python/ directory run the following command to install our custom perceptionloomo package pip install -e .

In case you have problems with the installation of opencv using pip you can try to build it directly from source using the following procedure.

mkdir depencency 
cd dependency
wget https://github.com/opencv/opencv/archive/3.4.5.zip
unzip 3.4.5.zip
mkdir opencv 
cd opencv-3.4.5
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=../../opencv -DFORCE_VTK=ON -DWITH_TBB=ON -DWITH_V4L=ON -DWITH_OPENGL=ON -DWITH_CUBLAS=ON -DCUDA_NVCC_FLAGS="-D_FORCE_INLINES" ..
make -j4
make install

Please find the following documents for an introduction to the Loomo robot and a socket protocol.

• Getting_Started_with_Loomo.pdf

• Environment_Setup_Robots.pdf

• Loomo_Deployment_Instruction.pdf

• Tutorial_Loomo_Race.pdf

• Loomo with Ros

• Loomo Follower App