Code Monkey home page Code Monkey logo

actiondetection-dbg's Introduction

Update

  • 2020.03.13: Release tensorflow-version and pytorch-version DBG complete code.
  • 2019.11.12: Release tensorflow-version DBG inference code.
  • 2019.11.11: DBG is accepted by AAAI2020.
  • 2019.11.08: Our ensemble DBG ranks No.1 on ActivityNet

Introduction

In this repo, we propose a novel and unified action detection framework, named DBG, with superior performance over the state-of-the-art action detectors BSN and BMN. You can use the code to evaluate our DBG for action proposal generation or action detection. For more details, please refer to our paper Fast Learning of Temporal Action Proposal via Dense Boundary Generator!

Contents

Paper Introduction

image

This paper introduces a novel and unified temporal action proposal generator named Dense Boundary Generator (DBG). In this work, we propose dual stream BaseNet to generate two different level and more discriminative features. We then adopt a temporal boundary classification module to predict precise temporal boundaries, and an action-aware completeness regression module to provide reliable action completeness confidence.

ActivityNet1.3 Results

image

THUMOS14 Results

image

Qualitative Results

Prerequisites

  • Tensorflow == 1.9.0 or PyTorch == 1.1
  • Python == 3.6
  • NVIDIA GPU == Tesla P40
  • Linux CUDA 9.0 CuDNN
  • gcc 5

Getting Started

Installation

Clone the github repository. We will call the cloned directory as $DBG_ROOT.

cd $DBG_ROOT

Firstly, you should compile our proposal feature generation layers.

Please compile according to the framework you need.

Compile tensorflow-version proposal feature generation layers:

cd tensorflow/custom_op
make

Compile pytorch-version proposal feature generation layers:

cd pytorch/custom_op
python setup.py install

Download Datasets

Prepare ActivityNet 1.3 dataset. You can use official ActivityNet downloader to download videos from the YouTube. Some videos have been deleted from YouTube,and you can also ask for the whole dataset by email.

Extract visual feature, we adopt TSN model pretrained on the training set of ActivityNet, Please refer this repo TSN-yjxiong to extract frames and optical flow and refer this repo anet2016-cuhk to find pretrained TSN model.

For convenience of training and testing, we rescale the feature length of all videos to same length 100, and we provide the 19993 rescaled feature at here Google Cloud or 微云. Then put the features to data/tsn_anet200 directory.

For generating the video features, scripts in ./tools will help you to start from scrach.

Testing of DBG

If you don't want to train the model, you can run the testing code directly using the pretrained model.

Pretrained model is included in output/pretrained_model and set parameters on config/config_pretrained.yaml. Please check the feat_dir in config/config_pretrained.yaml and use scripts to run DBG.

# TensorFlow version (AUC result = 68.37%):
python tensorflow/test.py config/config_pretrained.yaml
python post_processing.py output/result/ results/result_proposals.json
python eval.py results/result_proposals.json

# PyTorch version (AUC result = 68.26%):
python pytorch/test.py config/config_pretrained.yaml
python post_processing.py output/result/ results/result_proposals.json
python eval.py results/result_proposals.json

Training of DBG

We also provide training code of tensorflow and pytorch version. Please check the feat_dir in config/config.yaml and follow these steps to train your model:

1. Training

# TensorFlow version:
python tensorflow/train.py config/config.yaml

# PyTorch version:
python pytorch/train.py config/config.yaml

2. Testing

# TensorFlow version:
python tensorflow/test.py config/config.yaml

# PyTorch version:
python pytorch/test.py config/config.yaml

3. Postprocessing

python post_processing.py output/result/ results/result_proposals.json

4. Evaluation

python eval.py results/result_proposals.json

Citation

If you find DBG useful in your research, please consider citing:

@inproceedings{DBG2020arXiv,
  author    = {Chuming Lin*, Jian Li*, Yabiao Wang, Ying Tai, Donghao Luo, Zhipeng Cui, Chengjie Wang, Jilin Li, Feiyue Huang, Rongrong Ji},
  title     = {Fast Learning of Temporal Action Proposal via Dense Boundary Generator},
  booktitle   = {AAAI Conference on Artificial Intelligence},
  year      = {2020},
}

Contact

For any question, please file an issue or contact

Jian Li: [email protected]
Chuming Lin: [email protected]

actiondetection-dbg's People

Contributors

chengjie-cjwang avatar lijiannuist avatar linchuming avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

actiondetection-dbg's Issues

When will train.py be released?

Thanks for sharing the project!
Recently, I was learning DBG papers and codes, really want to try training with my own dataset. May i ask the time schedule of releasing the traing codes (train.py)?

Thumos code

Hello! Thanks for your work.could you send me your code on Thumos14 please?
it will help me a lot. thank you very much!!!
my email address : [email protected]

run error

Thanks for your work. But I met an error when I run this code:python pytorch/test.py config/config_pretrained.yaml .The error :
Traceback (most recent call last):
File "pytorch/test.py", line 11, in
from model import DBG
File "/mnt/songyan/10519_xuminhuang/ActionDetection-DBG-master/pytorch/model.py", line 4, in
from custom_op.prop_tcfg_op import PropTcfg
File "/mnt/songyan/10519_xuminhuang/ActionDetection-DBG-master/pytorch/custom_op/prop_tcfg_op.py", line 4, in
import prop_tcfg_cuda
ImportError: /mnt/songyan/10519_xuminhuang/anaconda3/lib/python3.7/site-packages/prop_tcfg_cuda-0.0.0-py3.7-linux-x86_64.egg/prop_tcfg_cuda.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __cudaRegisterFatBinaryEnd

my tool:
pytorch1.1
cuda9
python3.7
gcc5.4
Can someone help me solve this problem?thanks

Can you release the THUMOS14 features?

Hello! Thanks for your work, I have been looking for the features extracted from THUMOS14 dataset. While I haven't seen anyone released it. I wanna kown if you can release the features, which will help me a lot.

How to use C3D extracted features?

Thank you for your open source work. Optical flow feature extraction is too slow and does not have engineering significance. I want to try the features extracted by C3D. Your paper mentioned this. What are the details?

Extracting Feature Error

Hi, I just downloaded the feature from Google Drive.

And when use tar to extract features, it looks like:
tar: This does not look like a tar archive

Do you know there is something wrong with the feature?

I downloaded twice from the Google Drive.

Awesome project! Question about rescale the feature length.

Thanks for sharing!
The inference speed is extremely faster with a better result.
I'm thinking about how to reproduce the result on other datasets.
Could you share some detail or insight about how to use linear interpolation to rescale the feature length of all videos to same length 100 ?

Any method function, reference papers or codes will be thankful.

THUMOS14

Thank you a lot. Recently, our research group is following your work. If you can share the code and files of THUMOS14 with us, it will be convenient for us to do comparative experiments

About the provided features!!

Thank you for your work. The features of ActivityNet are also provided in the BSN code. Is there any difference between the features you provide?

[run error] No OpKernel was registered to support Op 'PropTcfg' with these attrs

Hi,
I had follow the instructions to installed tf1.9.0 and python3.6, and compiled "Proposal Feature Generation Layer" successfully, which generate "prop_tcfg.so". but when run "bash auto_run.sh", following errors were occured:

loading config file: ./config.yaml
loading config file: ./config.yaml
2019-11-23 17:13:54.840138: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-11-23 17:13:54.841548: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
Runing DBG model ...
0%| | 0/296 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/anaconda2/envs/tf19/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/anaconda2/envs/tf19/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/anaconda2/envs/tf19/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'PropTcfg' with these attrs. Registered devices: [CPU], Registered kernels:
device='GPU'; T in [DT_FLOAT]

 [[Node: model/PropTcfg = PropTcfg[T=DT_FLOAT, center_num=16, end_num=8, mode=0, start_num=8](model/strided_slice)]]

It seems like the Op 'PropTcfg' was not registered correctly, but How can I fix it? Thank you!

Question about PFG layer

Brilliant work and thanks for the open source code.

I'm now reading the paper and code about the DBG model and I have a question about the PFG layer.

According to the paper and the code, the parameter w_l and w_r for the the PFG layer's output at location (t_s, t_e, n, c), which corresponding to the w, h, t, c in the code if I have understood correctly, can be directly calculated under the formula (1)(2)(3)(4) in the "Proposal feature generation layer" section. But both the paper and code shows that this layer is a trainable layer. So will the w_l and w_r be updated during the backward? Since it looks like a variable that don't need to be trained rather than a trainable parameter to me and I'm confused about this. Could you please explain this? Thank you very much!

features csv different from provided

thanks for ur great work!

we used tools/run.sh to get videos' rgb and flow feature csvs,
and then used tools/data_process.py to join then together
but we found the joined csvs are different from the data u provided
any suggestion ? thanks!

the zip file contains three csvs
*_t.csv temporal
*_s.csv spatial
*_a.csv joined
ca.zip

Was my result correct?

After I downloaded the 14.2 GB ActivityNet features (Very hard, must use Tencent Weiyun VIP in my network environment), I ran the auto_run.sh and get the following results:

 [INIT] Loaded annotations from validation subset. 
 processoNumber of ground truth instances: 7292
 processoNumber of proposals: 472700 
Fixed threshold for tiou score: [0.5  0.55 0.6  0.65 0.7  0.75 0.8  0.85 0.9  0.95] 
[RESULTS] Performance on ActivityNet proposal task.
        Area Under the AR vs AN curve: 47.82460230389468%
AR@1 is          0.27855183763027974
AR@5 is          0.3704470652770159
AR@10 is         0.4054306088864509
AR@100 is        0.5950630828304992

@lijiannuist
I wonder if the result i got above is correct....?Why does it look like that the result is not good? Thanks...

proposal

Can someone give me a proposal.txt or proposal.json of DBG?My Laboratory Conditions can't make it. If you can generate proposal,can you send me one? Thanks.
My email is [email protected]

THUMOS14 code can you share ?

I admire your experiment very much,but I have encountered some troubles when trying to implement the algorithm related to Thumos14.
can you send me your code and the features on Thumos14?
my email address : [email protected]

Segmentation fault when define the whole model with PFG layer compiled

I have compiled the PFG layer to prop_tcfg.so. After that, i'm going to train a new network in which T == 500, for long videos. However, when i define the model, "Segmentation fault" breakouted, and this is the all error description.
Could u please tell what the problem is? Thanks!

error:
WARNING:tensorflow:From ***************/DBG/model.py:19: conv1d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.keras.layers.Conv1D instead.
WARNING:tensorflow:From ****************/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
Segmentation fault

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.