dlpbc / keras-kinetics-i3d Goto Github PK

keras implementation of inflated 3d from Quo Vardis paper + weights

License: MIT License

Python 100.00%

deep-learning computer-vision action-recognition pretrained-kinetics-model

keras-kinetics-i3d's Introduction

keras-kinetics-i3d

Keras implementation (including pretrained weights) of Inflated 3d Inception architecture reported in the paper Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset.

Original implementation by the authors can be found in this repository.

Sample Data (for Evaluation)

Similar to the original implementation, we evaluate the keras models using the RGB sample and Optical Flow sample (processed from video data) provided in the repository of the authors (see the data/ directory). Details about the preprocessing techniques applied to the data are specified in the authors' repository.

Usage

python evaluate_sample.py

or

[For help]
python evaluate_sample.py -h

With default flags settings, the evaluate_sample.py script builds two I3d Inception architecture (2 stream: RGB and Optical Flow), loads their respective pretrained weights and evaluates RGB sample and Optical Flow sample obtained from video data.

You can set flags to evaluate model using only one I3d Inception architecture (RGB or Optical Flow) as shown below:

# For RGB
python evaluate_sample.py --eval-type rgb

# For Optical Flow
python evaluate_sample.py --eval-type flow

Addtionally, as described in the paper (and the authors repository), there are two types of pretrained weights for RGB and Optical Flow models respectively. These are;

RGB I3d Inception:
- Weights Pretrained on Kinetics dataset only
- Weights pretrained on Imagenet and Kinetics datasets
Optical Flow I3d Inception:
- Weights Pretrained on Kinetics dataset only
- Weights pretrained on Imagenet and Kinetics datasets

The above usage examples loads weights pretrained on Imagenet and Kinetics datasets. To load weight pretrained on Kinetics dataset only add the flag --no-imagenet-pretrained to the above commands. See an example below:


# RGB I3d Inception model pretrained on kinetics dataset only
python evaluate_sample.py --eval-type rgb --no-imagenet-pretrained

Requirements

Keras
Keras Backend: Tensorflow (tested) or Theano (not tested) or CNTK (not tested)
h5py

License

All code in this repository are licensed under the MIT license as specified by the LICENSE file.
The i3d (rgb and flow) pretrained weights were ported from the ones released Deepmind in this repository under Apache-2.0 License

keras-kinetics-i3d's People

Contributors

Stargazers

Watchers

Forkers

willdamon bityangke ccanel kinect59 lvaleriu doople1st banuselintosun adrianmfi seleucia vanshaw2017 lfsblack bkcline vnigade socialcore-primary mkzirncz1 cameroncruz erdongchendou sharan99 chenmoshushi bsaghafi wangyu jiangdizhao nram812 issamhammad heinrich-heine ann-korea joeytang3377 xbeeprince alonsopg officework1993 devanshb26 abdkhanstd nik123 zhixiangwang-cn saurabhkumar8112 adewopova panlizhi jahsylla mounirb hkx97 beng-ern hanh30 noakawa steven-lemoal

keras-kinetics-i3d's Issues

Failure in building model when using custom classes.

I was trying to train on the UCF101 dataset as toy example. Fail to build model.
Following minimalist code snippet fails.

from keras_kinetics_i3d.i3d_inception import Inception_Inflated3d
i3d_model = Inception_Inflated3d(weights='rgb_imagenet_and_kinetics', endpoint_logit=False, classes=102, include_top=False)

Error given

  File "/home/hollowgalaxy/Github/experiments/video_extraction_experiments.py", line 76, in <module>
    i3d_test()
  File "/home/hollowgalaxy/Github/experiments/video_extraction_experiments.py", line 74, in i3d_test
    i3d_model = Inception_Inflated3d(weights='rgb_imagenet_and_kinetics', classes=102, include_top=False)
  File "/home/hollowgalaxy/Github/keras_kinetics_i3d/i3d_inception.py", line 515, in Inception_Inflated3d
    h = int(x.shape[2])
TypeError: __int__ returned non-int (type NoneType)

Thank you!

Hello @dlpbc ,

I just wanted to thank you for porting the I3D network to Keras. I used it to build a Sign Language Recognition prototype: https://github.com/FrederikSchorr/sign-language

Kind regards, Frederik

PS: In https://github.com/FrederikSchorr/sign-language/blob/master/model_i3d.py I added a function add_i3d_top that creates new top layers depending on the number of output classes.

Separable inception

Have you experimented with the separable version of i3d?

data is missing

/v_CricketShot_g04_c01_rgb.npy /v_CricketShot_g04_c01_flow.npy is missing

Converting the tf model to keras

Hello,

Great work, thanks!
I was just curious about how you converted the tf models and weights to Keras. Is there a simple way to do that?

Best,
Florian

How sensitive is the model to the flow method?

In your opinion, how sensitive is the model to the flow method?

I've tried it with some other flow methods on cuda (for speed) and it gives me absurd results. I also can't seem to re-train it on other flow methods. So I"m wondering if the model is very sensitive to the optical flow method and how would you deal with it if you needed something faster?

Any plans to support kinetics 600 weights?

Hi, love this work! Just wondering if you're planning to support any models on the updated kinetics 600 dataset?

how to test model on my sample video

Hi dears,
sorry for send email directly,

I'm interested in I3D method for action recognition and i cloned the github project "https://github.com/deepmind/kinetics-i3d"
i want to test the model with my sample recorded video but don't know how to convert my *.mp4 video file (25fps rgb) to *rgb.npz and *flow.npz

could it possible to help or share me a reprocessing script file

best regards.

Training

Hi dlpbc, your i3d keras model is awesome!

I am working on sign language recognition for deaf-mute-people, and need to finetune the provided model/weights on another video-dataset.

In case you have also trained/finetuned with I3D - Could you share a training/finetuning script? I am in particular interested in data_generator needed for fitting with data augmentation.

Thank you! br Frederik