Test data leaked into meta training about maml-pytorch HOT 9 CLOSED

NO.
in meta-learning setting, meta-training includes training and testing, meta-test includes training and testing as well. The label space between meta-training and meta-testing is separated properly. For sub-stage, say training and testing in meta-train or meta-testing, the leakage is normal, not an error.

from maml-pytorch.

Thank you for your response!
I am aware that the sub-stage update is achieved through fast_weights. My question is, at the end of the forward() function, does self.meta_optim.step() update parameters of the base learner, i.e., the \theta in the paper?

from maml-pytorch.

Yes, the meta_optim.step() will update \theta parameters only. The updates of fast_weights will be ignored in sub-stage of test.

from maml-pytorch.

At the meta-testing phrase, we need to forward() the meta-test tasks to evaluate the corresponding testing performance at the meta level (not the task level). More importantly, we only want to get the fast weights for those tasks, because the meta test tasks should not be used to update the meta learner, i.e., \theta. However, the \theta is also updated since self.meta_optim.step() is always part of forward().

Do you think there is any need to take out self.meta_optim.step() in meta testing?

from maml-pytorch.

In stead of self.meta_optim.step(), shouldn't we do the following?

if training is True: self.meta_optim.step()

from maml-pytorch.

Oh, maybe u r right.
During the test, no need to optimize the \theta parameters.
U can try it and see any problems.

from maml-pytorch.

I have already tried and it turned out the optimization step is required to be left out in meta testing. Otherwise it would give overly optimistic results for be trained on meta test sets.

from maml-pytorch.

Ok, thx. Feel free to submit a PR.

from maml-pytorch.

Hi, all. Please git pull to get the latest version which solved test data leaking bugs.

from maml-pytorch.