sj-li / ms-tcn2 Goto Github PK

MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation (TPAMI 2020)

License: MIT License

Python 97.59% Shell 2.41%

ms-tcn2's Introduction

MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation (TPAMI 2020)

This repository provides a PyTorch implementation of the paper MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation.

Environment

Python3, pytorch

Training:

Download the data folder, which contains the features and the ground truth labels. (~30GB) (If you cannot download the data from the previous link, try to download it from here)
Extract it so that you have the data folder in the same directory as main.py.
To train the model run sh train.sh ${dataset} ${split} where ${dataset} is breakfast, 50salads or gtea, and ${split} is the split number (1-5) for 50salads and (1-4) for the other datasets.

Evaluation

Run sh test_epoch.sh ${dataset} ${split} ${test_epoch}.

Cite:

@article{li2020ms,
   author={Shi-Jie Li and Yazan AbuFarha and Yun Liu and Ming-Ming Cheng and Juergen Gall},
    journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
    title={MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation}, 
    year={2020},
    volume={},
    number={},
    pages={1-1},
    doi={10.1109/TPAMI.2020.3021756},
}

@inproceedings{farha2019ms,
  title={Ms-tcn: Multi-stage temporal convolutional network for action segmentation},
  author={Farha, Yazan Abu and Gall, Jurgen},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={3575--3584},
  year={2019}
}

ms-tcn2's People

Contributors

Stargazers

Watchers

ms-tcn2's Issues

Length of raw videos is different from the extracted features

Hi, I'd like to do the process on raw videos but found that the length of raw videos and labels (downloaded from the official Breakfast dataset) are different from the extracted I3D features and the corresponding labels.

Did you do any preprocessing on the raw videos before extracting features?

Questions about fine-tuning dataset

Hi,

First, thanks for such a great contribution in MSTCNN++.
In the paper, you have mentioned that you tried the fine-tuning of GTEA dataset.
I was wondering that did you tried fine-tuning Breakfast dataset?
If yes, can you share the features or the experiments of it?
If not, is it possible to share some materials on how I can get the fine-tuning Breakfast dataset.

Thank you

cannot reproduce paper results with default configuration

We are using docker image “nvcr.io/nvidia/pytorch:22.03-py3” from NVIDIA
git clone https://github.com/sj-li/MS-TCN2.git
cd MS-TCN2
ln -s $data_path dat
# dataset in breakfast, 50salads, gtea
# split in (1-5) for 50salads and (1-4) for the other datasets
python3 main.py --action=train --dataset=$dataset --split=$split --num_epochs=100 --num_layers_PG=11 --num_layers_R=10 --num_R=3

The results in paper which is in TABLE 16:

And all the results mismatch with paper somehow:

	F1 0.10	F1 0.25	F1 0.50	Edit	Acc
50salads split 1	73.4	70.1	60.2	64.2	76.7
50salads split 2	72.2	69.2	59.8	64.1	81.7
50salads split 3	71.5	68.4	62.3	66.3	75.2
50salads split 4	79.3	77.0	69.1	71.6	83.3
50salads split 5	80.2	77.8	72.6	71.5	86.7
gtea split 1	80.1	77.4	64.6	71.5	74.4
gtea split 2	83.7	81.0	70.1	76.6	78.3
gtea split 3	90.6	89.1	79.2	88.2	79.2
gtea split 4	87.9	85.6	70.5	83.1	74.9

Are these mismatch reasonable, or did I miss any configuration?

Training parameter problem

How many epochs did you run in each dataset? I see in the code that the number of epoch you set is 100. Is this inconsistent with the original ms-tcn?

mask in DilatedResidualLayer

DilatedResidualLayer class in model.py lacks input: mask?

Could you provide a (pre)-trained checkpoint file?

Problems in training

After the training, I did not save any training results when I tried the test. Could you please tell me the reason? At last, thank you very much for the code you provided.

Performance Results

Hi,

Can I just check whether the results quoted in the paper are based on the training or test splits? I am getting ball park results for the training, however finding the test to be quite a bit lower.

Thanks

Performance variance

Hi Li,

Many thanks for sharing the code and I really like this interesting work. I ran the code on the data your shared, but the performance seems slightly different from the reported in the table. For example, this is the average result for the 5 splits I obtained on 50salads, "Acc:82.03, Edit:72.14, F1@10: 79.51, F1@25: 77.59, F1@50: 69.45". I was wondering what I missed in the reproduction.

Best regards
ja

sj-li / ms-tcn2 Goto Github PK

ms-tcn2's Introduction

MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation (TPAMI 2020)

Environment

Training:

Evaluation

Cite:

ms-tcn2's People

Contributors

Stargazers

Watchers

Forkers

ms-tcn2's Issues

Recommend Projects

Recommend Topics

Recommend Org