Detecting Human-Object Interactions in Real-Time
This is the repo for a open project detecting human object interactions in real-time, see more detail on our Tech Report.
Contents
Requirements
Hardware
GPU: Titan, Titan Black, Titan X, K20, K40, K80, GTX
Software
You should install matlab to validate the training result of HOI-RT. You should install cuda, opencv and cudnn. Then set the 1-3 line of Makefile:
GPU=1
CUDNN=1
OPENCV=1
Installation
- Clone the HOI-RT repository
Firstly, make a new folder named detection and thencd detection && git clone --recursive [email protected]:lmingyin/HOI-RT.git
- Build the project
cd $HOI-RT && make -j8
- Load the trained model
Load the trained model which has been trained on vcoco and our labeled dataset. And put the model in the detection folder.
Test
After successful installation, now you can test HOI-RT.
cd $HOI-RT/
./darknet detector test cfg/vcoco.data cfg/yolo-vcoco608.cfg ../yolo-vcoco608_80000.weights data/kick.jpg
Train
Load coco and vcoco datasets
V-COCO dataset builds off MS COCO, please download MS-COCO images and annotations(coco 2014 is enough), make sure all which in a new folder coco, the downloaded extracted image folders like train2014, val2014, test2014 should in the new folder images which under coco, the downloaded extracted annotations like instances_train2014.json, instances_val2014.json should in the new folder annotations which under coco.
- Clone V-COCO repository (recursively, so as to include COCO API).
cd coco git clone --recursive https://github.com/s-gupta/v-coco.git
- Pick out annotations from the COCO annotations.
cd v-coco python script_pick_annotations.py coco-data/annotations
- Build
cd $VCOCO_DIR/coco/PythonAPI/ && make cd $VCOCO_DIR && make
- Show the V-COCO label
Copy vcoco_show.py from script folder to the folder v-coco and runcd v-coco python vcoco_show.py
- Get more details
See more V-COCO introduction in V-COCO Repository.
Load our dataset
More training data should be loaded from our dataset1 and our dataset2 , then combine the two folder to RelationDataset and put it under the folder detection.
Generate training labels
- Make labels from V-COCO dataset
Copy vcoco_label.py from script folder to v-coco folder and thenFinally, in folder filelist will generate a file trainVCOCO.txt. And in folder coco will outputs a folder named labels which contain all training labels.cd coco && mkdir filelist cd v-coco && python vcoco_label.py
- Make labels from our dataset
Copy voc_relation_label.py from the script folder to the detection folder, and thenFinally, trainOurs.txt will be generated in current folder, and training labels will be generated in every action folder in RelationDataset.cd detection && python voc_relation_label.py
Merge two training labels
Copy trainOurs.txt to the folder filelist. And then
cd filelist && cat trainVCOCO.txt trainOurs.txt > train.txt
Load the pretrained model
Load the pretrained model, and put it in the detection folder
Train the model
Before training, you should set cfg/vcoco.data
train = Your_Path/coco/filelist/train.txt
then use the following command to train the model
cd ROI-RT/
make clean && make -j8
./darknet detector train cfg/vcoco.data cfg/yolo-vcoco608.cfg ../darknet19_448.conv.23
Validate
Validate action detection on APagent
- Generate the test labels
Copy vcoco_test_action.py from the script folder to v-coco folder, and thenYou will get a folder vcoco_action_valid. Put it in the detection folder.cd v-coco python vcoco_test_action.py
- Validate the model on action detection
open vcoco.data and set
cd HOI-RT/cfg/
and then validate the modelvalid = Your_Path/coco/filelist/vcoco_test.txt eval =
in the current folder a folder results will be generated, you should put it in the folder vcoco_action_valid, and thencd HOI-RT/ ./darknet detector valid cfg/vcoco.data cfg/yolo-vcoco608.cfg backup/yolo-vcoco608_80000.weights
run the script validate_action.m, you will get the APagent for every action.cd HOI-RT/matlab
Validate relation detection on AProle
- Generate the test labels
Copy vcoco_test_relation.py from the script folder to v-coco folder, and thenA folder vcoco_relation_valid will be generated, and put it in the folder detection.python vcoco_test_relation.py
- Validate the model on relation detection
open vcoco.data and set
cd HOI-RT/cfg/
and then validate the modelvalid = Your_Path/coco/filelist/vcoco_test.txt eval = relation
in current folder a folder results will be generated, you should put it in the folder vcoco_relation_validcd HOI-RT/ ./darknet detector valid cfg/vcoco.data cfg/yolo-vcoco608.cfg backup/yolo-vcoco608_80000.weights
run the script validate_relation.m, you will get the AProle for every action.cd HOI-RT/matlab