Codes have been restructured and ready for benchmarking on datasets.

This repository contains implementation of the paper Video-Pose Embedding For Activities of Daily Living (VPN) in Keras. VPN works with a base 3D video understanding model such as Inception3D for feature extraction and adds Pose based Attention Network to weight the importance of video features towards activity recognition.

We show the results of VPN on four activity recognition datasets - Smarthomes, NTU-60, NTU-120 and NUCLA for different evaluation protocols. Currently, in this repo only I3D backbone is supported.

Before the start of VPN training, following steps should be completed

• Create a new or use the existing configuration files stored in config folder. The configuration files are specified by the type of model and the dataset to use. Refer to args defined in the main.py file for more details.

• Specify the paths of following files needed as input for VPN in the config yaml file.

  • Skeleton : 3D pose data stored as npz files for each video clip
  • CNN : RGB video data
  • Splits : Training, Validation and Test video data splits

• Make sure the necessary folders for storing model weights are created.

• Currently, only NTU60 and NTU120 is supported and config files and related files will be updated for other datasets Smarthomes, NUCLA later.

To train VPN with I3D as backbone on NTU60, execute the below line.

python main.py --dataset ntu60

• Reorganize codebase to get started with model training quickly
• Add support for Smarthomes and NUCLA datasets
• Benchmark results
• Update results
• Upload Trained models
• Upload Demo videos for all datasets
• Add support for other base 3D video models

@misc{das2020vpn,
    title={VPN: Learning Video-Pose Embedding for Activities of Daily Living},
    author={Srijan Das and Saurav Sharma and Rui Dai and Francois Bremond and Monique Thonnat},
    year={2020},
    eprint={2007.03056},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}