Code Monkey home page Code Monkey logo

backtal's Introduction

Background-Click Supervision for Temporal Action Localization

This repository is the official implementation of BackTAL. In this work, we convert existing action-click supervision to the background-click supervision and develop a novel method, called BackTAL. Paper from arXiv or IEEE.

Illustrating the architecture of the proposed BackTAL

Requirements

To install requirements:

conda env create -f environment.yaml

Data Preparation

Download

Download pre-extracted I3D features of Thumos14, ActivityNet1.2 and HACS dataset from BaiduYun with code back.

Please ensure the data structure is as below
├── data
   └── Thumos14
       ├── val
           ├── video_validation_0000051.npz
           ├── video_validation_0000052.npz
           └── ...
       └── test
           ├── video_test_0000004.npz
           ├── video_test_0000006.npz
           └── ...
   └── ActivityNet1.2
       ├── training
           ├── v___dXUJsj3yo.npz
           ├── v___wPHayoMgw.npz
           └── ...
       └── validation
           ├── v__3I4nm2zF5Y.npz
           ├── v__8KsVaJLOYI.npz
           └── ...
   └── HACS
       ├── training
           ├── v_0095rqic1n8.npz
           ├── v_62VWugDz1MY.npz
           └── ...
       └── validation
           ├── v_008gY2B8Pf4.npz
           ├── v_00BcXeG1gC0.npz
           └── ...
     

Background-Click Annotations

The raw annotations of THUMOS14 dataset are under directory './data/THUMOS14/human_anns'.

Evaluation

Pre-trained Models

You can download checkpoints for Thumos14, ActivityNet1.2 and HACS dataset from BaiduYun with code back. These models are trained on Thumos14, ActivityNet1.2 or HACS using the configuration file under the directory "./experiments/". Please put these checkpoints under directory "./checkpoints".

Evaluation

Before running the code, please activate the conda environment.

To evaluate BackTAL model on Thumos14, run:

cd ./tools
python eval.py -dataset THUMOS14 -weight_file ../checkpoints/THUMOS14.pth

To evaluate BackTAL model on ActivityNet1.2, run:

cd ./tools
python eval.py -dataset ActivityNet1.2 -weight_file ../checkpoints/ActivityNet1.2.pth

To evaluate BackTAL model on HACS, run:

cd ./tools
python eval.py -dataset HACS -weight_file ../checkpoints/HACS.pth

Results

Our model achieves the following performance:

threshold 0.3 0.4 0.5 0.6 0.7
mAP 54.4 45.5 36.3 26.2 14.8
threshold average-mAP 0.50 0.75 0.95
mAP 27.0 41.5 27.3 4.7
threshold average-mAP 0.50 0.75 0.95
mAP 20.0 31.5 19.5 4.7

Training

To train the BackTAL model on THUMOS14 dataset, please run this command:

cd ./tools
python train.py -dataset THUMOS14

To train the BackTAL model on ActivityNet v1.2 dataset, please run this command:

cd ./tools
python train.py -dataset ActivityNet1.2

To train the BackTAL model on HACS dataset, please run this command:

cd ./tools
python train.py -dataset HACS

Citing BackTAL

@article{yang2021background,
  title={Background-Click Supervision for Temporal Action Localization},
  author={Yang, Le and Han, Junwei and Zhao, Tao and Lin, Tianwei and Zhang, Dingwen and Chen, Jianxin},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2021},
  publisher={IEEE}
}

Contact

For any discussions, please contact [email protected].

backtal's People

Contributors

vividle avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

backtal's Issues

about frame_num

大佬想问一下,在 dataset.py 文件中__getitem__中返回的frame_num是什么含义,似乎不是视频的片段数。🤣

Cannot find "WtalDataset"

Hi, thanks a lot for your work!
in file "eval.py",line 18
from dataset.dataset import WtalDataset
I cannot find the WtalDataset from dataset

ActivityNet Click Lable

I really admire your work. Can I get files similar to ActivityNet1.2_Background-Click-Annotation_A1.txt?

about DDP

Hello, Can I use DistributedDataParallel in this code? After I add the code of DistributedDataParallel, the performance will drop a lot, and it is useless to increase the learning rate or reduce the batch size, can you give me some suggestions? Thank you very much.

about background click

Thank you for your excellent work!But I have some questions about the use of background click information. In the code, except for score separation module and affinity module, I can't find the use of click_label anywhere else. So I want to know how to implement the experiment that includes background click but removes these two modules in the ablation experiment. There must be something I missed here. Can you give me some help? Thank you very much.

about BEOID

您能否提供BEOID和activitynet 1.2的背景标注文件呢?

Feature extraction

Hi, thanks a lot for your work!

Can you share the code used to extract the I3D feature??

I want to understand how to extract the features for RGB frames and optical flow.

Thank you!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.