Hi @chandru4ni ,

Thank you for your reply. In my case the problem was I have two python versions, 2.7 and 3.5 installed in my system and mistakenly I have had installed protobuf 3.4.0 for python 2.7. When I removed it and installed protobuf 2.6.1 for python 2.7 everything works fine.

Hope this will help someone in the future.

Thank you @chandru4ni and @LisaAnne for your help.

from eccv2016.

We have seen the json output and it contains the labels for each image in the data set.

Now we want to train the model with our data set and as per our inspection we need three files for training the model.

1. data/vocab.txt
2. data/descriptions_bird.train_noCub.fg.json
3. data/description_sentence_features.p

As of now you have stored these files in your path and using them and allowing us to download. But since want to train the model for our dataset we want to know how to generate the description_sentence_features.p file. Please suggest.

Is there anything we need to have before we train your model with our data set? We will continue to look for details from our end.

from eccv2016.

from eccv2016.

Thanks Lisa.

Yes you are right that description_sentence_features.p is a label embedding for different classes (200 X 1000 numpy array). You have told to use one of the hot vectors for this. If I am not wrong, I understand from your paper that you are using Compact Bilinear Pooling to get this vector. Though I spent some time on inspecting the code base I was not able to find the code to generate this vector even for your dataset.

So I need your help here to understand the flow since I am planning to use my own data set.

regards,
Chandra.

from eccv2016.

Is my understanding that description_sentence_features.p is representation of features (1000) for 200 classes of birds correct?

from eccv2016.

from eccv2016.

Thanks Lisa.
I saw the code of extract_val.py and yes it is generating the label embeddings and saving the file as pickle file. But while I ran the script to generate the pickle file, it is crashing. While debugging I observed that it is crashing while processing through the net at net.forward() line in extract_val.py.

Below are my few observations regarding make_class_embedding.sh script.
Below are the inputs to extract_val.py.
image_net=examples/ECCV2016/prototxt/deploy.prototxt
#word_net=examples/ECCV2016/prototxt/wtd_1000_all.prototxt
word_net=examples/ECCV2016/prototxt/wtd_1000.prototxt

Chandra

#model=snapshots/description_1006
model=gve_models/description_1006

I see that word_net file name is wtd_1000_all.prototxt and it does not exist in prototxt directory as part of downloaded files. But a file named wtd_1000.prototxt file is in prototxt directory, so I have changed to that file name. And the model filename is snapshots/description_1006 but I do not see any file in snapshots directory. And description_iter_1000.cafemodel is generated when we run train_description.sh in snapshots directory. We have description_1006.caffemodel downloaded from your site at gve_models. So I have changed the model to gve_models/description_1006.caffemodel

So I wanted to check if these changes are causing the crash. If yes, are we then missing wet_1000_all.prototxt and snapshots/description_1006.caffemodel files?

If not, I will continue debugging the code to see the cause of the crash. I request your thoughts on above and if possible you can throw light on the cause of the crash.

regards,
Chandra.

from eccv2016.

So my question if the changes i have done are causing the crash? And I am unable to find the file names which you have given in the make_class_embedding.sh. Are they missing in your code base by any chance? If possible you can let me know reason of the crash for any other reason. I will continue debugging the code for further analysis.

regards,
Chandra.

from eccv2016.

Since I have changed the file names in make_class_embedding.sh as below, i want to confirm before I proceed with further debugging. The lines in bold are the original lines and below them are modified by me. As the original files are not available to compare I want your inputs in this regard.

#word_net=examples/ECCV2016/prototxt/wtd_1000_all.prototxt
word_net=examples/ECCV2016/prototxt/wtd_1000.prototxt

#model=snapshots/description_1006
model=gve_models/description_1006

regards,
Chandra.

from eccv2016.

I changed the script as below and used wtd_1000_20words.prototxt file rather than wtd_1000_all.prototxt file which you had used in the script. With this make_class_embedding.sh ran successfully.
word_net=examples/ECCV2016/prototxt/wtd_1000_20words.prototxt

I generated the model with generated label embeddings using train_explanation_label.sh. But when I tried to use this generated model for for evaluation script eval_explanation-label.sh, I got all captions as same for all birds and that it is not some symbols as given below.

"196.House_Wren/House_Wren_0057_187157.jpg", "caption": "" & "."}, {"image_id": "099.Ovenbird/Ovenbird_0070_93140.jpg", "caption": "" & "."}, {"image_id": "135.Bank_Swallow/Bank_Swallow_0018_129891.jpg", "caption": "" & "."}, {"image_id": "117.Clay_colored_Sparrow/Clay_Colored_Sparrow_0068_110706.jpg", "caption": "" & "."}

Looks like something wrong in the prototxt file I used or the steps I am following. Please advice with your suggestion.

regards,
Chandra.

from eccv2016.

Corrected the comments which had some typos.

I changed the script as below and used wtd_1000_20words.prototxt file rather than wtd_1000_all.prototxt file in make_class_embedding.sh script. With this make_class_embedding.sh ran successfully.
word_net=examples/ECCV2016/prototxt/wtd_1000_20words.prototxt

I generated the model with label embeddings using train_explanation_label.sh. But when I tried to use this generated model for for evaluation script eval_explanation-label.sh, I got all captions as same for all bird images and that it has got some symbols rather than words as given below.

"196.House_Wren/House_Wren_0057_187157.jpg", "caption": "" & "."}, {"image_id": "099.Ovenbird/Ovenbird_0070_93140.jpg", "caption": "" & "."}, {"image_id": "135.Bank_Swallow/Bank_Swallow_0018_129891.jpg", "caption": "" & "."}, {"image_id": "117.Clay_colored_Sparrow/Clay_Colored_Sparrow_0068_110706.jpg", "caption": "" & "."}

Looks like something wrong in the prototxt file I used or the steps which I am following. Please advice with your suggestion.

regards,
Chandra.

from eccv2016.

Hi Lisa,

Now I am getting the captions correctly by using the make_class_embedding.sh script and using the deploy.prototxt and wtd_1000.prototxt files. And this script requires snapshots/description_1006 model to be available. But if we want to run this script for the first time then we will not have the model available, hence as you indicated earlier I wanted to generate the label embeddings using word2vec or Grove method. The output of your make_class_embedding.sh script is 200X1000 numpy array and I assume 200 corresponds to classes of birds. In that case word2vec and Grove provides word embeddings. Hence I am thinking on how to get the class embeddings required for our explanation model. Any suggestions from your end will be helpful.

regards,
Chandra.

from eccv2016.

Hi Lisa,

Now I am getting the captions correctly by using the make_class_embedding.sh script and using the deploy.prototxt and wtd_1000.prototxt files. And this script requires snapshots/description_1006 model to be available. But if we want to run this script for the first time then we will not have the model available, so I thought of generating the label embeddings using word2vec or Grove method. The output of your make_class_embedding.sh script is 200X1000 numpy array and I assume 200 corresponds to classes of birds. In that case using word2vec and Grove we get word embeddings and not mapped to classes. Hence I am thinking on how to get the class embeddings required for our explanation model, should I look for other tools? I will continue exploring from my end, just wrote to you so that any suggestions from your end would be helpful.

regards,
Chandra.

from eccv2016.

from eccv2016.

Thanks Lisa, yeah I am setting the data set to train the model. In the mean time I found that right now the code is able to provide visual explanation for all birds that belong to bilinear_pred.p master data set. If I give a new bird which does not belong to master data set then it is not identifying it correcty. Is it a prerequisite for the project to have the bird belong to only to this data set, since your research is more specific to provide visual explanations rather the classifying the image or identifying the image. But when I go for my data set, I need to first preprocess the image to get the descriptors to identify the image. Right now the current code is not working for preprocessing the image. Let me know if my understanding is correct so that I will continue my exploration on getting my image preprocessed to get descriptors through the image net.

Regards,
Chandra.

from eccv2016.

Thanks Lisa, yeah I am working on the data set to train the model and generate the embeddings as you indicated above. In the mean time I found that right now the code is able to provide visual explanation for all birds that belong to bilinear_pred.p master data set. If I give a new bird which does not belong to master data set then it is not identifying it correctly. Is it a prerequisite for the project to have the bird belong to only to this data set, since your research is more specific to provide visual explanations rather the classifying the image or identifying the image. But when I go for my data set, I need to first preprocess the image to get the descriptors to identify the image. Right now the current code is not working for preprocessing the image. Let me know if my understanding is correct so that I will continue my exploration on getting my image preprocessed to get descriptors through the image net.

Regards,
Chandra.

from eccv2016.

Hi Lisa,
Finally we are able to use your visual explanation code component for fight data set. Below are few customizations we have done. Need to check your feedback on the same.

The .p file which is used for training we have generated it using Doc2vec model of genim. We just feed in one sentence for each class and generate a 1X1000 vector or sentence embedding corresponding to each class.

We were able to generate the .p using make_class_embedding.sh script. We first gave one shot vector as input during training in explanation-label.prototxt file as below. (10 corresponds to number of classes)
'size_onehot' : 10

And tried to use the above trained model in make_class_embedding.sh to generate .p with average of probabilities per class. But when we use this .p for training it was giving 'nan' for loss value during the 20 iteration itself. Tried changing value of size_onehot but did not help. We require your help here. What is the value of size_onehot we need to use in prototxt file and what does it correspond to?

Since make_class_embedding.sh generated .p did not work for us we used the sentence embedding vector generated by Doc2vec model and trained the explanation-label model and saw reducing loss.

We were able to successfully train and evaluate the flight dataset with good accuracy.

We want your feedback on using .p and if you can help with your advice on using size_onehot in prototxt file.

Thanks in advance.

from eccv2016.

We are now looking into discriminative loss model of yours. This has multiple layers and looks complex. We will keep you updated on this.

regards,
Chandra.

from eccv2016.

Hi --

Yes, the discriminative loss is a bit tricky, especially in Caffe (to backprop through REINFORCE you need to be a bit creative with the existing layers).

It seems like your understanding of size_onehot is correct. It is possible that the sentence embedding for Doc2vec is better for your problem, so I would not worry about this too much. I played around with different embeddings (including just a one hot vector indicating a class) and the results were all pretty similar.

from eccv2016.

Oh ok. That's seems convincing :-). Thanks.

Your work on discriminative loss is very interesting, we will try to explore from our end and will get back for any help.

from eccv2016.

Hi @chandru4ni

I'm trying to run Lisa's code on a different dataset. But when I tried I got the error "This program requires version 3.4.0 of the Protocol Buffer runtime library, but the installed version is 2.6.1.". Did you also run into this error? If so may I please know how you could resolve it?

I hope you follow the steps given by Lisa and could train the model.

Thank you in advance!

from eccv2016.

from eccv2016.

from eccv2016.