We propose a Dual-Attention Network (DANet) aims to learn robust video representation for view-invariant action recognition. The DANet is composed of relation-aware spatiotemporal self-attention and spatiotemporal cross-attention modules. The relation-aware spatiotemporal self-attention module learns representative and discriminative action features. This module captures local and global long-range dependencies, as well as pairwise relations among human body parts and joints in the spatial and temporal domains. The cross-attention module learns view-invariant attention maps and generates discriminative features for semantic representations of actions in different views.

The main function and modules source code will be released for future work and to facilitate communication

There are 3 datasets to download:

• NTU RGB+D 60 Skeleton
• NTU RGB+D 120 Skeleton
• UESTC skeleton Datatset

NTU RGB+D 60 and 120 Dataset Request dataset here: http://rose1.ntu.edu.sg/Datasets/actionRecognition.asp Download the skeleton datasets:

• nturgbd_skeletons_s001_to_s017.zip (NTU RGB+D 60)
• nturgbd_skeletons_s018_to_s032.zip (NTU RGB+D 120)
• Extract above files to ./data/nturgbd_raw

UESTC Dataset Request dataset here: https://github.com/HRI-UESTC/CFM-HRI-RGB-D-action-database

Put downloaded data into the following directory structure:

• data/
  • UESTC/ ... # raw data of UESTC
  • ntu/
  • ntu120/
  • nturgbd_raw/
    • nturgb+d_skeletons/ # from nturgbd_skeletons_s001_to_s017.zip ...
    • nturgb+d_skeletons120/ # from nturgbd_skeletons_s018_to_s032.zip

Generating Data:

cd ./data/ntu # or cd ./data/ntu120
 # Get skeleton of each performer
 python get_raw_skes_data.py
 # Remove the bad skeleton 
 python get_raw_denoised_data.py
 # Transform the skeleton to the center of the first frame
 python seq_transformation.py

Here we present the attention results learned by DANet on the NTU-60 dataset, where the ellipses' radius are proportional to the joint relevance. The greenlight ellipses indicate the correct attention weights, with a larger radius corresponding to higher weights.

• DANet Attention weights (camera 1 (45 degree view).

• DANet Attention weights( camera 3 (side view)).

• DANet Attention weights (Drinking water)

• DANet Attention weights (Hugging other person)

• DANet Attention weights (Hand-shaking).

This repo is based on 2s-AGCN and ST-TR. The data processing is borrowed from SGN

Thanks to the original authors for their work!

For any questions, feel free to contact: [email protected]