visionlearninggroup / r-c3d Goto Github PK

when training with provided caffe3d found the error:

F1220 15:06:31.091182 8949 blob.hpp:140] Check failed: num_axes() <= 4 (5 vs. 4) Cannot use legacy accessors on Blobs with > 4 axes.

Hi all,

I follow the same pipeline to download the videos and extract all the frames. However, when I try to train the model, I found the ``train_data_3fps_flipped.pkl" file is missing. Does anyone here know how to generate or download this file ?

Hi Huijuan,

I find that your README has explained everything well. I really appreciate it!

However, when I generate the pickle data for training and testing R-C3D model, I checked everything in your repo, but can't find meta file like 'validation_set.mat' and the error is shown as follow:

IOError: [Errno 2] No such file or directory: '${DATA_PATH}/data/val/validation_set_meta/validation_set.mat'

Could you please upload it or tell me the hints to find it, or how to generate it? Thanks for your help! @huijuan88

Best,
June

Hi, I test the caffemodel and the results are saved to log file. But I have no idea what is the matrix below?
I notice that in test.py cls_win and cls_score are stacked, however, there are many spaces between two floating points. Could you please explain it?
Thanks for your kindness!

activity: 12
[[ 43.073074 183.75291 0.9771269 ]
[242.433 376.31635 0.7799468 ]
[346.45883 455.46292 0.67514086]
[118.58941 263.16885 0.60216105]]

Thanks a lot for sharing the code.
When i test your released R-C3D model on THUMOS'14 dataset, the mAP is lower than yours. The results are as follows:
Fixed threshold for tiou score: [ 0.1 0.2 0.3 0.4 0.5]
mean AP at different thresholds: [ 0.44844552 0.42947421 0.40517887 0.36456814 0.29133984]
The [email protected] is simillar, but the others are bad. I don't know why. could you give me some suggestions? I'm looking forward to your reply. Thank you very much!

Hi,@huijuan88
I got a compile error when I run make -j8 && make pycaffe
Could you tell me how to solve this problem?

i can't find convolution3d_layer.cpp in caffe3d folder.

Hi,@huijuan88
I know the inference speed of R-C3D can reach 569 fps. However, the training time wasn't mentioned in the paper. Now I have only one 1080Ti GPU. So I'm not sure how long the training would cost.
Could you tell me your devices and training time ?
Thanks for your attention.

As said in your paper, you sample frames at 3 fps, and each clip contain 768 frames. I have tried something like this, but find many videos are shorter than 768 frames after sampling. How do you handle this issue ? Padding zero array?
Hope to receive you reply.

Hey,

Has anybody succeeded in training it on custom dataset. I need to know what things to change in the training code for it to run on custom dataset.

Or is it possible if I arrange my dataset in activitynet format and run activitynet's training code?

Any help would be appreciated.

Thanks

 I run the model half a month ago, and that i just works very well. But strange happened today( i need to attend the other class so leave it for two weeks), the output log just have activity like below. May somebody use my computer to install some strange package? I have no idea now. i guess i need find the bug. But somebody here give me a clue, it would be very nice. thanks everyone

activity: 167
[[3.4108795e+02 5.9973071e+02 7.1076835e-03]
[2.0220493e+02 3.6329410e+02 7.1076835e-03]
[5.3852698e+02 6.8188672e+02 7.1076835e-03]
[9.1723251e-01 7.4949814e+01 7.1076835e-03]
[6.6526924e+01 2.0988675e+02 7.1076835e-03]
[3.0652689e+02 4.4988675e+02 7.1076835e-03]
[1.3052692e+02 2.7388675e+02 7.1076835e-03]
[6.8126740e+02 7.6669012e+02 7.1076835e-03]
[6.0252698e+02 7.4588672e+02 7.1076835e-03]
[5.1857452e+00 4.2374323e+02 7.1076835e-03]]

I have already downed the "activitynet_iter_135000.caffemodel", but when I prepare to generate the prediction log file on the validation set, and run "./experiments/activitynet/test/script_test.sh" , some questions occured,it said "Waiting for ./experiments/activitynet/snapshot/activitynet_iter_135000.caffemodel to exist...", and trapped in a dead circle. I can't understand this question, could you please explain it?
Thanks for your kindness!

Hi @huijuan88,

Could I share your trained well models for thumos'14 and ActivityNet v1.3 datasets in your R-C3D paper [1]? Thank you very much.

[1] R-C3D: Region Convolutional 3D Network for Temporal Activity Detection

Hi, @huijuan88

Thanks for your great work - CMS-RC3D. I have noticed in your paper (page 7, Section 4.4 ):

"We re-train the origin RC3D model with better hyper-parameter fine-tuning as our baseline model (i.e. akin to the single-scale version of our model), and we achieve better results for the original RC3D compared to the results reported in [33] (about 7.0% improvement @Map 0.5; refer to the 12th and 15th rows of Table 4) "

My question is what hyper-parameters in R-C3D that you have modified?

Looking forward to your reply.

Best wishes,
Guang Chen.

hi,
I find a question,but I am not sure whether is right or not.
in the code :activitynet_log_analysis.py

left_frame = float(l.split()[1])*stride + start_frame
right_frame = float(l.split()[2])*stride + start_frame

and then i change the code to :

     left_frame = float(l.split()[0].split('[')[-1])*stride + start_frame
     right_frame = float(l.split()[1])*stride + start_frame

and then,it works ,or it will come across some error ,and I hope you can check it.
otherwise, there still another question:

activity:  80
[[19.049221  65.19444    0.1244299]]
activity:  115
[[9.4955711e+00 7.9366196e+01 6.5591358e-02]]

what's the meaning of the three parameters??
thanks for your attention.

hi,thanks for providing your code.When i training the model on my own dataset,i just see the number of iteration and i don't know the number of epoches.So,how can i know it?
when i training on my own dataset,my loss picture just like the picture below. is it right? i got Map 16.43% on my own dataset which including seven gestures？i think it is two low,how can i make some changes to improve the results? thanks for your reading

Hey,

Thanks for sharing the code. I am playing the code with activitynet example. I have 1080 ti GPU with 11G memory. With that, I can only run your code with length=196. I just wonder any tricks there to make it run with length=786? I would really appreciate your help!

Hi, I am running the code and find that it takes about two hours every 1000 iterations on Tesla M40, and I noticed that the default iteration is 60k totally...so it may cost so long time to finish training...
What is the reason for that? And what is your original training time?
Thanks!

Thanks a lot for sharing the code.
I am confused about this model?(activitynet_iter_30000_3fps.caffemodel)
a. It is a classification model or detection model?
b. how to get this model?
Thank you in advance!

Hi,

Your code is very helpful. However, i have a question about the pre-trained sports_1m model. When I try to train R-C3D on the pre-trained sports_1m model, it prints out the following error:

F0420 12:21:31.234062   550 blob.cpp:479] Check failed: count_ == proto.data_size() (1728 vs. 0) 
*** Check failure stack trace: ***

real	0m26.785s
user	0m3.368s
sys	0m23.920s

I've tried the models for both C3D-v1.0 and C3D-v1.1. I was wondering which pre-trained model you used and if you would be able to provide that? Thanks!

Hi, Thank you so much for sharing this repo. Would you please publicise the scripts for thumos dataset?

Where is the SPP layer?

firstly,thank you for your sharing the code.
when I test your released R-C3D model on activitynet dataset,the mAP and the proposal task 
performance are much lower than yours or the expected result.The results are as follows:

[INIT] Loaded annotations from validation subset.
Number of ground truth instances: 7220
Number of predictions: 29
Fixed threshold for tiou score: [ 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95]
[RESULTS] Performance on ActivityNet detection task.
Average-mAP: 4.16666666667e-05
[INIT] Loaded annotations from validation subset.
Number of ground truth instances: 7220
Number of proposals: 29
Fixed threshold for tiou score: [ 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95]
[RESULTS] Performance on ActivityNet proposal task.
Area Under the AR vs AN curve: 0.0164542936288%
I don't know what lead to these abnormal results. could you give me some suggestions? I'm looking forward to your reply. Thank you very much!

Hello, In your code ,I find that the anchor size is set to [2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56] (THUMOS14),but in the paper , the anchor size is [2,4,5,6,8,9,10,12,14,16],and then I should modify this parameter to train the network? @huijuan88

It seems that the 1st-5th group won't be changing weights during the training since there is a pre-trained model in your description below:
"The C3D model weight pretrained on Sports1M and finetuned on ActivityNet dataset"

Is this a correct understanding?
Can you provide a more comprehensive expalantion about "end-to-end", in terms of training via back-propagation? For instance, maybe we can use your paper's Fighure 2 as a reference.
Thanks a lot!

thanks for your job!
now i want to finetune the 135000 caffemodel by using myself dataest,and because of network ,i can't download ActivityNet dataest videos,so i don't know dataest format
i really hope you can help me.thx again!

Thanks a lot for providing the open-source code!

Could you provide the pre-processing file for R-C3D?

Highly appreciate your time and help!

The training data is my own, and the config file is default ,when I trained it

rpn: num_positive 15
rpn: num_negative 49
I0306 13:32:56.740681 15943 accuracy_layer.cpp:96] Accuracy: 0.90625
I0306 13:32:56.741325 15943 accuracy_layer.cpp:101] Class 0 accuracy : 0.877551
I0306 13:32:56.741331 15943 accuracy_layer.cpp:101] Class 1 accuracy : 1
TRAIN
I0306 13:32:56.769326 15943 solver.cpp:228] Iteration 97, loss = 0.509901
I0306 13:32:56.769354 15943 solver.cpp:244] Train net output #0: accuarcy = 0.956522
I0306 13:32:56.769363 15943 solver.cpp:244] Train net output #1: loss_cls = 0.1252 (* 1 = 0.1252 loss)
I0306 13:32:56.769368 15943 solver.cpp:244] Train net output #2: loss_twin = 0.0680313 (* 1 = 0.0680313 loss)
I0306 13:32:56.769371 15943 solver.cpp:244] Train net output #3: rpn_accuarcy = 0.90625
I0306 13:32:56.769376 15943 solver.cpp:244] Train net output #4: rpn_accuarcy_class = 0.877551
I0306 13:32:56.769378 15943 solver.cpp:244] Train net output #5: rpn_accuarcy_class = 1
I0306 13:32:56.769382 15943 solver.cpp:244] Train net output #6: rpn_cls_loss = 0.306048 (* 1 = 0.306048 loss)
I0306 13:32:56.769387 15943 solver.cpp:244] Train net output #7: rpn_loss_twin = 0.0106217 (* 1 = 0.0106217 loss)
I0306 13:32:56.769392 15943 sgd_solver.cpp:106] Iteration 97, lr = 0.0001

just like this,it stoped here, then there is nothing more.
I wonder what's wrong with it,

thanks very much

can someone please post their makefile.config? i can install caffe normally but when it comes to this repo make -j8 is giving me errors.

Thanks for sharing your code.

I have several problems when I tested your released model ( activitynet_iter_135000.caffemodel ) on ActivityNet.

1. the Average-mAP is always 0.
2. when I use a larger mini-batch, 768, same as you, it is always out of memory. I noticed that you use Titan X Maxwell in your paper, its memory size is 12G. I used a P100 GPU, which has 16G memory.

update:
add a testing result

frames : [[ 0 0 657 8]]
bg_name : ./preprocess/activityNet/frames/validation/UgXPt2LydrY
fg_name : ./preprocess/activityNet/frames/validation/UgXPt2LydrY
flipped : False

wins: (roi proposal)

[[ 0. 255. ]
[ 66.44532776 255. ]
[ 0. 218.65629578]
[ 33.66741943 255. ]
[ 0. 187.37054443]
[ 91.33938599 255. ]
[ 0. 152.22647095]
[ 110.46815491 255. ]
[ 129.6413269 255. ]
[ 7.7613945 103.98934937]
[ 40.77736282 135.82566833]
[ 160.34649658 255. ]
[ 82.78195953 214.23434448]
[ 98.47563934 229.97964478]
[ 33.91320801 165.2225647 ]
[ 72.57516479 167.88267517]
[ 66.27030945 197.79762268]
[ 56.43632507 151.39079285]
[ 50.34194946 182.03378296]
[ 102.55069733 149.96676636]
[ 17.97701263 149.67550659]
[ 104.42219543 200.01075745]
[ 0. 88.10974884]
[ 24.20291519 119.38748169]
[ 136.2828064 232.16848755]
[ 1.62534332 133.79647827]
[ 88.35385132 183.80404663]
[ 118.39078522 165.66413879]
[ 86.41411591 133.68508911]
[ 120.50378418 215.75134277]
[ 70.2829361 117.98705292]
[ 134.58239746 182.00643921]
[ 215.72177124 255. ]
[ 199.02253723 244.72337341]
[ 37.98304749 85.67688751]
[ 0. 119.01252747]
[ 152.89222717 248.92503357]
[ 7.99191284 236.2943573 ]
[ 54.12044907 101.61773682]
[ 150.3513031 197.65653992]
[ 182.44004822 228.91236877]
[ 177.01557922 255. ]
[ 100.27285004 193.0748291 ]
[ 186.86495972 255. ]
[ 143.31906128 255. ]
[ 71.65406036 220.22286987]
[ 55.12468719 204.090271 ]
[ 166.38612366 213.62513733]
[ 39.10673523 188.43531799]
[ 23.66375732 172.69458008]
[ 87.56541443 236.63554382]
[ 84.17002106 177.20883179]
[ 68.50379944 161.24717712]
[ 132.28718567 225.0355072 ]
[ 116.0929718 209.18348694]
[ 30.61087036 226.16656494]
[ 14.54845428 210.32543945]
[ 0. 72.69412231]
[ 199.42425537 255. ]
[ 0. 57.3568573 ]
[ 52.17543793 145.33273315]
[ 21.40447998 68.92416382]
[ 46.54234314 242.66990662]
[ 36.53536606 129.28849792]
[ 87.35679626 205.07170105]
[ 71.47348022 188.49401855]
[ 119.4033432 237.06689453]
[ 102.90635681 220.75663757]
[ 56.15686035 172.87768555]
[ 231.45487976 255. ]
[ 148.17895508 241.41604614]
[ 39.5399971 156.67909241]
[ 0. 98.71743011]
[ 23.49741364 141.02084351]
[ 5.47562981 51.86833191]
[ 11.60011292 167.71418762]
[ 7.35813904 125.4258728 ]
[ 105.71432495 143.67492676]
[ 52.45306396 228.57843018]
[ 59.79278564 215.7227478 ]
[ 89.90881348 127.5004425 ]
[ 122.04985809 159.56503296]
[ 185.95800781 223.4786377 ]
[ 137.89151001 175.58615112]
[ 73.58313751 111.59540558]
[ 154.01879883 191.46774292]
[ 202.06903076 239.9201355 ]
[ 7.11604309 143.87721252]
[ 169.75775146 207.30343628]
[ 53.36356354 136.12887573]
[ 69.6763382 152.30215454]
[ 57.7285347 95.62669373]
[ 41.7742691 79.62611389]
[ 165.89404297 248.00344849]
[ 85.56309509 167.89205933]
[ 0. 50.21304321]
[ 114.11366272 229.84934998]
[ 0. 63.91038132]
[ 0. 35.81119156]
[ 90.43177032 173.78588867]]

scores for each roi (the prob for bg class is 0.99):

[[ 9.94664431e-01 1.44916630e-05 1.25025894e-04 ..., 3.92486945e-05
6.97385576e-06 8.20186506e-06]
[ 9.99880195e-01 3.64945294e-07 4.93661491e-06 ..., 8.19759180e-07
1.41763380e-07 1.39852929e-07]
[ 9.98758435e-01 3.27796079e-06 4.09543391e-05 ..., 6.80767016e-06
1.65701283e-06 2.54305928e-06]
...,
[ 9.99230146e-01 2.16727585e-06 3.06134789e-05 ..., 3.57966314e-06
9.80683694e-07 1.68363363e-06]
[ 9.99356925e-01 1.81754160e-06 2.93088888e-05 ..., 2.77124832e-06
7.97892653e-07 1.47988487e-06]
[ 9.99994516e-01 2.24784671e-08 5.76837465e-07 ..., 3.10989847e-08
8.70350814e-09 1.38918095e-08]]

Hi,
i want to know which Cuda and CUDNN version this model worked on ??

It is mentioned that your activity detection speed can reached at 569 fps. I wonder how you compute the speed? I guess it include the time of post-processing such as NMS, right? Then, does it include the time of frames reading and pre-processing before input to the C3D becuse it is usually time-consuming? Hope to receive your reply soon.

Hi @huijuan88
Thank you for updating the code of thumos14 and charades!
I want to run the training on thumos14, however I found the ucf101 fine-tuned model provided by C3D v1.0 is not compatible with your code.
And I could not find the newest version (C3D-v1.1) caffe model fine-tuned on ucf101.
Could you plz tell me how can I find the v1.1 model or provide us your pretrained model for thumos14?
Thanks a lot!

For training the model for a new dataset, how to generate the .mat file . (validation_set.mat for thumos14, for example)

Hi, Huijuan @huijuan88
I am using a card of 1080Ti with 11G memory, but 2.5G was used by other students, so I was only left with 8.5G memory with GPU. But when I run the test script in ActivityNet with your provided script, only loaded one 1 video's frams(768 images), but out of memory at the step:
blobs_out = net.forward(**forward_kwargs)
"""
F0713 15:08:15.452706 22317 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
Aborted (core dumped)
"""

so could you plz tell me what is your GPU type and how many GPUs have you used when testing and training this code?
Thanks in advance!

/usr/bin/ld: cannot find -ltatlas /usr/bin/ld: cannot find -lsatlas

hi in the training code filter_roidb is usd but no such module is imported
where could I find this function?

Can you share the code used to generate this groundtruth JSON?
My result was OK with THUMOS14 MATLAB eval code, but zero mAP uses following command:

python ./Evaluation/get_detection_performance.py ./thumos14_gt_test.json ./results.json

source

thumos14_gt_test.json file

Thanks a lot!

Hi! When I read the code in generate_roidb_training.py, I found something weird.

According to this part, the WINS should be a list contains only one element: 768*8

According to this part, if start = 0, then end = 768*8, then start and end would be written into roidb:

While in minibatch.py, the code create the data according to this start and end:

However, the video size is 3x512x112x112 according to cfg.TRAIN.LENGTH = (512, )

Is there anything wrong since 512 definitely less than 768 * 8?

thank you for sharing your code for video action detection! I hava test the code in thumos14 dataset and found the strange result,all the activity index is 0(background).Could you give me some hints ?Thank you!

Thanks for your great contribution!
And I have one question about the window size for generating roi in generate_roidb_512.py

As shown above, you create a sliding window whose size is 512 to collect the data used for training, what confusing me is whether this parameter a empirical parameter? Or it is used after analysis on the thumos dataset?

from tdcnn.config import cfg
from tdcnn.twin_transform import clip_wins, twin_transform_inv
and so on, in code test can not import

Hi, @huijuan88 ， thank you for sharing your code!
In the paper "R-C3D: Region Convolutional 3D Network for Temporal Activity Detection", it is said the each video is divided into several segments of 768 frames, and R-C3D only produces detection results which belong to the segment inputed into the network.Is it necessary to Integrate the detection results of each segment to get the detection results for the complete videos in order to calculate the mAP? If yes, how do you achieve this?
Any help will be appreciated !

Thanks a lot for sharing the code.
It works well for generate_roidb_train.py, but when generating the validation using
$python generate_roidb_validation.py

It has the following errors:
Generate Classes
0
1
2
../.././preprocess/activityNet/frames/validation/tTEAlDsmZrA/image_-0001.jpg
Traceback (most recent call last):
File "generate_roidb_validation.py", line 82, in
val_roidb = generate_roidb('validation')
File "generate_roidb_validation.py", line 57, in generate_roidb
tmp = generate_roi(vid, start, end, stride, split)
File "generate_roidb_validation.py", line 37, in generate_roi
raise
TypeError: exceptions must be old-style classes or derived from BaseException, not NoneType

Any suggestions?
Thanks again!

when i run the script of "script_train.sh":
I got a AttributeError: 'NoneType' object has no attribute 'astype'
the whole like this:

I0328 11:24:09.393389 27646 net.cpp:761] Ignoring source layer loss
Solving...
Traceback (most recent call last):
  File "./experiments/thumos14/train_net.py", line 97, in <module>
    max_iters=args.max_iters)
  File "/****/R-C3D/experiments/thumos14/../../lib/tdcnn/train.py", line 159, in train_net
    model_paths = sw.train_model(max_iters)
  File "/****/R-C3D/experiments/thumos14/../../lib/tdcnn/train.py", line 100, in train_model
    self.solver.step(1)
  File "/****/Deeplearning/R-C3D/experiments/thumos14/../../lib/roi_data_layer/layer.py", line 124, in forward
    blobs = self._get_next_minibatch()
  File "/****/R-C3D/experiments/thumos14/../../lib/roi_data_layer/layer.py", line 48, in _get_next_minibatch
    return get_minibatch(minibatch_db, self._num_classes)
  File "/****/R-C3D/experiments/thumos14/../../lib/roi_data_layer/minibatch.py", line 31, in get_minibatch
    video_blob = _get_video_blob(roidb, random_scale_inds)
  File "/****/R-C3D/experiments/thumos14/../../lib/roi_data_layer/minibatch.py", line 72, in _get_video_blob
    cfg.TRAIN.CROP_SIZE, random_idx)
  File "/****/R-C3D/experiments/thumos14/../../lib/utils/blob.py", line 32, in prep_im_for_blob
    im = im.astype(np.float32, copy=False)
AttributeError: 'NoneType' object has no attribute 'astype'

Does someone else have this problem? please help,thanks a lot

Once I train and test the model, How to use the model for getting the detection result for new input video? Lets say, I have trained it for my custom dataset and evaluated the performance. How to deploy it for detection.

Hello,

thank you for sharing your code.
The function filter_roidb() is not available or imported in the lib/tdcnn/train.py but called at line 117 of train.py
could you please provide this function or shall I just ignore it by commenting this line?

Hi @huijuan88,

Could I share your validation_set.mat for thumos'14 dataset? Thank you very much.

META_FILE='${PATH}/data/val/validation_set_meta/validation_set.mat'