Measure the model performance about maml-pytorch HOT 17 OPEN

@HongduanTian Yes, also it is happening every 500 episodes. Did you use original code from here. Does that have model performance part ?

Thank you,
KK

from maml-pytorch.

Tasks in testing phase have the same structures as those in training phase. So, in testing phase, query_set should be generated as what in training does. The optimal theta will adapt on support data in testing tasks and the performance will be measured on query data in testing tasks.

Just treat the task as data in common supervised learning. The structure of the 'data' should keep consistent.

@HongduanTian Hi, have you ever tried saving the model, and load model weights in an extra evaluate code? I have tried, I got ~47% accuracy when training with this repo(miniImagenet_train.py), but when I tried the operation mentioned above, I can only get ~44% accuracy. Can you provide some insights to me? Many thx!

from maml-pytorch.

Hi,
I also notice that there is no measure of the model performance. And in the code, there is another problem that the evaluation is performed on the test set, which is performed on val set in MAML.

Have you deal with such problem?

Thank you,
Tian

from maml-pytorch.

Yep, but in MAML original code, you have to set --train=False, --test_set=True after finishing training the model so that you can measure the performance. Now I am trying to modify this pytorch version in sight of MAML.
If you have dealt with the problem, could you please share your version with me ?

Thank you.
Tian

from maml-pytorch.

Yes. Sure. I will give a try and update you.

Thank you,
KK

from maml-pytorch.

@HongduanTian So in the original code, were you able to fine-tune your trained model with less number of samples (lets say only 5 samples for that new class) ?

Thank you,
KK

from maml-pytorch.

from maml-pytorch.

@HongduanTian Thank you so much for the reply.

I am using this repository (dragen1860 pytorch implementation), trying to write a inference code to predict the new images using the saved model.

I am saving the complete model self.net in meta.py.

Added a condition in miniimagenet_train.py to train and test the model.

However I am not successful in predicting a image from the save model.

I am getting NotImplementedError.

Have you tried the similar approach in pytorch version?

Thank you,
KK

from maml-pytorch.

from maml-pytorch.

Thank you for your response. And apologies for the delay in response.

Could you please tell me how did you predict the new images ?

Also lets say if my complete training dataset has 100 classes, and I am using n_way=4. In this case neural network model would be having only 4 output signals right? Kindly correct me if I am wrong.
How can we use this model to predict the images ?

Thank you,
KK

from maml-pytorch.

In main function, after you have trained the model, the parameters in model are already well trained. Thus, generate the tasks from test dataset folder, put them into maml model, and predict the test data with maml.fine_tuning() will be okay.

For the 2nd question, yes, you're right. If n_way=4, then you will obtain a tensor with the size of ( ,4)output.

from maml-pytorch.

In main function, after you have trained the model, the parameters in model are already well trained. Thus, generate the tasks from test dataset folder, put them into maml model, and predict the test data with maml.fine_tuning() will be okay.
After I train the model, I get a model with common theta right ? And when I do maml.fine_tuning() theta of the model would be tuned as per new task right ? Are we giving less number of samples in fine_tuning(). I see that even test_dataset contains huge number of samples. (Kindly correct me if I am wrong)

For the 2nd question, yes, you're right. If n_way=4, then you will obtain a tensor with the size of ( ,4)output.
I am just wondering, I trained a model with 100 classes. And I get a model with (n_way = 4) 4 outputs. How can I use this model for predicting the images?

Thank you,
KK

from maml-pytorch.

Well, The theta after being trained is the model parameters which can adapt new tasks fast. You can view the theta as the optimal. When you fine-tuning the model on new tasks, the theta should only be changed during inner loop phase so that it can adapt the new task. But the initial theta for each new task is same and will not be changed after predicting the tasks.

In MAML, the test tasks number is 600, and I choose this number as well.

And I am confused with your words ''I trained a model with 100 classes. And I get a model with (n_way = 4) 4 outputs.'' Do you mean you train the model with 100 classes classification tasks and the output is 4? If you just trained the model with 4 classes classification tasks, the output of the network is 4. When you test your model, you just need to generate tasks from test dataset with the same structure(like 4 classes k shot) in training tasks, and images predictions can be performed.

English isn't my native language, so my explanations may sometimes confuse you or I misunderstand your questions. I am sorry for that if I don't express my answer clearly. If your questions are not well solved, you can ask me again.

Best,
Tian

from maml-pytorch.

@HongduanTian Sorry for the delayed response.

Actually I understood the concept wrongly, you have explained it correctly.

So initial training is to get only the common theta model which can adapt to new classes. And in testing we pass only k samples to fine-tune the inner loop. And get new theta model which is capable to predicting classes in the new task.

And how about the query_set during the testing phase ? Do I need to have only k number of samples of a class or should I have in n_query number of samples for the same class ?

Thank you,
KK

from maml-pytorch.

Tasks in testing phase have the same structures as those in training phase. So, in testing phase, query_set should be generated as what in training does. The optimal theta will adapt on support data in testing tasks and the performance will be measured on query data in testing tasks.

Just treat the task as data in common supervised learning. The structure of the 'data' should keep consistent.

from maml-pytorch.

@HongduanTian Thank you again for your explanation and patience. I really appreciate that.
I am trying to understand the MAML architecture and how actually the code works.
I will be back with more doubts. Hope that is ok?

Thank you,
KK

from maml-pytorch.

from maml-pytorch.