BoT-SORT: Robust Associations Multi-Pedestrian Tracking

Nir Aharon, Roy Orfaig, Ben-Zion Bobrovsky NVIDIA Jetson Orin AGX opitimisation in the dev cycle, only for experimental useage.

https://arxiv.org/abs/2206.14651

The goal of multi-object tracking (MOT) is detecting and tracking all the objects in a scene, while keeping a unique identifier for each object. In this paper, we present a new robust state-of-the-art tracker, which can combine the advantages of motion and appearance information, along with camera-motion compensation, and a more accurate Kalman filter state vector. Our new trackers BoT-SORT, and BoT-SORT-ReID rank first in the datasets of MOTChallenge [29, 11] on both MOT17 and MOT20 test sets, in terms of all the main MOT metrics: MOTA, IDF1, and HOTA. For MOT17: 80.5 MOTA, 80.2 IDF1, and 65.0 HOTA are achieved.

The code was tested on Ubuntu 20.04

BoT-SORT code is based on ByteTrack and FastReID.
Visit their installation guides for more setup options.

Step 1. Create Conda environment and install pytorch.

conda create -n botsort_env python=3.7
conda activate botsort_env

Step 2. Install torch and matched torchvision from pytorch.org.
The code was tested using torch 1.11.0+cu113 and torchvision==0.12.0

Step 3. Install BoT-SORT.

git clone https://github.com/NirAharon/BoT-SORT.git
cd BoT-SORT
pip3 install -r requirements.txt
python3 setup.py develop

Step 4. Install pycocotools.

pip3 install cython; pip3 install 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'

Step 5. Others

# Cython-bbox
pip3 install cython_bbox

# faiss cpu / gpu
pip3 install faiss-cpu
pip3 install faiss-gpu

Download MOT17 and MOT20 from the official website. And put them in the following structure:

<dataets_dir>
      │
      ├── MOT17
      │      ├── train
      │      └── test    
      │
      └── MOT20
             ├── train
             └── test

For training the ReID, detection patches must be generated as follows:

cd <BoT-SORT_dir>

# For MOT17 
python3 fast_reid/datasets/generate_mot_patches.py --data_path <dataets_dir> --mot 17

# For MOT20
 python3 fast_reid/datasets/generate_mot_patches.py --data_path <dataets_dir> --mot 20

Link dataset to FastReID export FASTREID_DATASETS=<BoT-SORT_dir>/fast_reid/datasets. If left unset, the default is fast_reid/datasets

Download and store the trained models in 'pretrained' folder as follow:

<BoT-SORT_dir>/pretrained

• We used the publicly available ByteTrack model zoo trained on MOT17, MOT20 and ablation study for YOLOX object detection.

• Ours trained ReID models can be downloaded from MOT17-SBS-S50, MOT20-SBS-S50.

After generating MOT ReID dataset as described in the 'Data Preparation' section.

cd <BoT-SORT_dir>

# For training MOT17 
python3 fast_reid/tools/train_net.py --config-file ./fast_reid/configs/MOT17/sbs_S50.yml MODEL.DEVICE "cuda:0"

# For training MOT20
python3 fast_reid/tools/train_net.py --config-file ./fast_reid/configs/MOT20/sbs_S50.yml MODEL.DEVICE "cuda:0"

Refer to FastReID repository for addition explanations and options.

By submitting the txt files produced in this part to MOTChallenge website and you can get the same results as in the paper.
Tuning the tracking parameters carefully could lead to higher performance. In the paper we apply ByteTrack's calibration.

• Test on MOT17

cd <BoT-SORT_dir>
python3 tools/track.py <dataets_dir/MOT17> --default-parameters --with-reid --benchmark "MOT17" --eval "test" --fp16 --fuse
python3 tools/interpolation.py --txt_path <path_to_track_result>

• Test on MOT20

cd <BoT-SORT_dir>
python3 tools/track.py <dataets_dir/MOT20> --default-parameters --with-reid --benchmark "MOT20" --eval "test" --fp16 --fuse
python3 tools/interpolation.py --txt_path <path_to_track_result>

• Evaluation on MOT17 validation set (the second half of the train set)

cd <BoT-SORT_dir>

# BoT-SORT
python3 tools/track.py <dataets_dir/MOT17> --default-parameters --benchmark "MOT17" --eval "val" --fp16 --fuse

# BoT-SORT-ReID
python3 tools/track.py <dataets_dir/MOT17> --default-parameters --with-reid --benchmark "MOT17" --eval "val" --fp16 --fuse

• Other experiments

Other parameters can be used without passing --default-parameters flag.
For evaluating the train and validation sets we recommend using the official MOTChallenge evaluation code from TrackEval.

# For all the available tracking parameters, see:
python3 tools/track.py -h 

cd <BoT-SORT_dir>
python3 tools/demo.py video -f yolox/exps/example/mot/yolox_x_mix_det.py -c yolox/pretrained/bytetrack_x_mot17.pth.tar --with-reid --fp16 --fuse --save_result

Our camera motion compensation module is based on the OpenCV contrib C++ version of VideoStab Global Motion Estimation, which currently does not have a Python version.
Motion files can be generated using the C++ project called 'VideoCameraCorrection' in the GMC folder.
The generated files can be used from the tracker.

In addition, python-based motion estimation techniques are available and can be chosen by passing
'--cmc-method' <files | orb | ecc> to demo.py or track.py.

• Create OpenCV VideoStab GMC python binding or write Python version.
• Add deployment code.

@misc{https://doi.org/10.48550/arxiv.2206.14651,
  doi = {10.48550/ARXIV.2206.14651},
  url = {https://arxiv.org/abs/2206.14651},
  author = {Aharon, Nir and Orfaig, Roy and Bobrovsky, Ben-Zion},
  keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {BoT-SORT: Robust Associations Multi-Pedestrian Tracking},
  publisher = {arXiv},
  year = {2022},
  copyright = {arXiv.org perpetual, non-exclusive license}
}

A large part of the codes, ideas and results are borrowed from ByteTrack, StorngSORT, FastReID and YOLOX. Thanks for their excellent work!