C3D accuracy/prob blob extraction has different results compared to accuracy computed by C3D fine_tuning test about c3d HOT 23 CLOSED

When you extract probability (prob blob), do you use shuffle: false?
If your shuffle: true, then you prob blobs are not in the same order with your input list.

from c3d.

No it is false. here it is the one I use for feature extraction:
layers {
name: "data"
type: VIDEO_DATA
top: "data"
top: "label"
image_data_param {
source: "/data/Atousa/ActivityNet/ActivityNet_valid_input.txt"
use_image: true
mean_file: "/home/atousa/project/C3D/examples/c3d_finetuning/ActivityNet_train16_128_mean.binaryproto"
batch_size: 80
crop_size: 112
mirror: false
show_data: 0
new_height: 128
new_width: 171
new_length: 16
shuffle: false
}
}

and for finetuning test:

layers {
name: "data"
type: VIDEO_DATA
top: "data"
top: "label"
image_data_param {
source: "/data/Atousa/ActivityNet/ActivityNet_valid_input.txt"
use_image: true
mean_file: "train01_16_128_171_mean.binaryproto"
use_temporal_jitter: false
batch_size: 50
crop_size: 112
mirror: false
show_data: 0
new_height: 128
new_width: 171
new_length: 16
shuffle: true
}
}

from c3d.

I am afraid that fc6/fc7 features that I am extracting are not correct either. I am using your matlab script to read blobs.

from c3d.

1. I saw you have two different mean files.
2. Minor: in fine-tuning, you can choose mirror: true for more data augmentation.

from c3d.

Unfortunately this is not the issue. The data layer I have sent you is not for finetuning. It is for testing which is provided to test the finetuned model. Using ActivityNet_train16_128_mean.binaryproto I still get 30% accuracy which I beleive this one is correct.

My issue is for on feature extraction that gives 0 accuracy.

Thanks!

from c3d.

Fro finetuning and features extraction I use the same mean_file: c3d_finetuning/ActivityNet_train16_128_mean.binaryproto

For testing I also tried with mean_file: ActivityNet_train16_128_mean.binaryproto I get 30% accuracy. But for feature extraction I get wrong accuracy and probabilities.

Is it possible that features extraction only need video frames [1...N] and does not work on chunk of videos e.g [100..200] , [1200,1400],.......?

Thanks!

from c3d.

I just realized that in your example for UCF101 feature extraction, you do not provide any label and for all labels you put 0. I guess your provided script extract_image_features.cpp by facebook does not correctly extract prob and accuracy layers that needs correct label? In my list I provide correct labels rather than only 0.

input/frm/v_ApplyEyeMakeup_g01_c01/ 1 0
input/frm/v_ApplyEyeMakeup_g01_c01/ 17 0
input/frm/v_ApplyEyeMakeup_g01_c01/ 33 0
input/frm/v_ApplyEyeMakeup_g01_c01/ 49 0
input/frm/v_ApplyEyeMakeup_g01_c01/ 65 0
input/frm/v_ApplyEyeMakeup_g01_c01/ 81 0
input/frm/v_ApplyEyeMakeup_g01_c01/ 97 0
input/frm/v_ApplyEyeMakeup_g01_c01/ 113 0
input/frm/v_ApplyEyeMakeup_g01_c01/ 129 0
input/frm/v_ApplyEyeMakeup_g01_c01/ 145 0
input/frm/v_BaseballPitch_g01_c01/ 1 0
input/frm/v_BaseballPitch_g01_c01/ 17 0
input/frm/v_BaseballPitch_g01_c01/ 33 0
input/frm/v_BaseballPitch_g01_c01/ 49 0
input/frm/v_BaseballPitch_g01_c01/ 65 0
input/frm/v_BaseballPitch_g01_c01/ 81 0

from c3d.

A few things to note:

1. Testing and feature extraction need to use the same mean file as when you do fine-tuning. That says if you use a.mean for fine-tuning, then you need to use that file for both testing and feature extraction.

2. "Is it possible that features extraction only need video frames [1...N] and does not work on chunk of videos e.g [100..200] , [1200,1400],.......?" -> as in the list file, we specify the starting frame, so it sould work for your case. you can specific starting frame from your available frames, e.g. 10 or 1200

3. For feature extraction, the labels play no role, meaning you can put whatever number. However, if you want to use the same list file for testing, then you need to have the true corresponding labels.

4. In testing, shuffle: true while feature extraction shuffle: false. Also, make sure that the prefix file has the same order with the list file (corresponding one-to-one), to make sure one input clip will give you exactly one feature-prefix (at the SAME ORDER)

from c3d.

About 3) if labels play no role, then I can not extract 'prob' and 'accuracy' using features_extraction since softmax/accuracy computation requires labels. Is that right?

About 1) and 4) I am doing correctly.

Thanks for your response!

from c3d.

you can use feature extraction tool to extract prob, then base on prob, you can compute accuracy. prob does not need labels.

from c3d.

Computing accuracy base on 'prob' by feature extraction is almost zero. I am doing doing step 1 and 2 correctly. I was thinking maybe your script does not support 2) which it seems this is not the case.

It is very strange that test accuracy are different from accuracy I compute based on 'prob' extracted from feature_extraction.

Best!

from c3d.

Hi @AtousaTorabi I did compute accuracy from prob (extracted prob) quite a few times and got consistent results. Please carefully check your part of code to make sure there is no bug. Thanks.

from c3d.

Hi @dutran
Thank you very much for your response. I have investigated prob generated from features_extraction. They are all pretty low and similar values for all classes. So even the probabilities seems not good. And accuracy computed based on them is 0.01 while test_net gives 0.3 accuracy. So one of them should not be correct.
I know it should be something wrong in the way I use feature_extraction. and based on these results fc6 features that I have extracted are also not good.

from c3d.

I think I found the issue. In fine-tuning the model prototxt that I use has the layers names "fc6", "fc7", "fc8". but the model prototxt that I was using for feature extraction has layers names "fc6-1", "fc7-1", "fc8-1". So I guess pretrained model was not copying correctly. Could this be the issue?

from c3d.

Yes, it is. What happened is you just copied the learned weights for conv-layers, but all fully connected layers are randomly initialized.

from c3d.

I see. Thanks a lot for your response. It is much appreciated! :)

from c3d.

Does it work for you after corrected those names?

from c3d.

Yes now it gives the same accuracy as the one computed by test_bin.
thanks a lot!

from c3d.

Glad to hear. Good luck with your experiments!

from c3d.

Hi Atousa @AtousaTorabi , I happened to came across this problem too. So I was wondering when you extract features, did you just use the names "fc6", "fc7" instead of "fc6-1", "fc7-1"? Because in the fine-tuned model, there is only fc6 and fc7 layer, no fc6-1 and fc7-1 exist. Is my understanding correct? Thanks.

from c3d.

Hi,

I have fine-tuned the C3D model on my dataset and used names "fc6", "fc7"
for fine-tuning. However for feature extraction (i.e. using my fine-tuned
model) I was using "fc6-1" "fc7-1" names by mistake which was initializing
the fully-connected layers 6 and 7 of model by random number rather than
using the weights that I have trained during fine-tuning. So you should use
the same feature layers names that you used in finetuning, then feature
extraction correctly copy all the trained weights rather than
initializing them.

I hope this help!
Best,
Atousa

On Thu, Oct 1, 2015 at 9:37 PM, yzloveyz [email protected] wrote:

Hi Atousa @AtousaTorabi https://github.com/AtousaTorabi , I happened to
came across this problem too. So I was wondering when you extract features,
did you just use the names "fc6", "fc7" instead of "fc6-1", "fc7-1"?
Because in the fine-tuned model, there is only fc6 and fc7 layer, no fc6-1
and fc7-1 exist. Is my understanding correct? Thanks.

—
Reply to this email directly or view it on GitHub
#35 (comment).

from c3d.

Hi @AtousaTorabi , thanks for your answer. It indeed helps, just the naming problem.

from c3d.

Hi Atousa @AtousaTorabi ，“conv3d_deepnetA_sport1m_iter_1900000” file 's FC layers' names are "fc6-1","fc7-1","fc8-1",is it right?Why don't we use the same name except the output layer to coppy the value from “conv3d_deepnetA_sport1m_iter_1900000” in the "c3d_finetuning" example?It will do better than random initial,isn't it?
But I find,in the "c3d_finetuning"example,the FC layers are different .Why?I'm confused.
Thanks.

from c3d.