Stage-1 pre-train with target feature about mmt HOT 2 CLOSED

Hi, thanks for the question.
The operation of "forward for target input" in pre-training can be thought of as a trick to improve the pre-training performance on the target domain. The insight is to roughly adapt the BN layers to the target domain, since the running mean&var in BN can take each input into the computation even there is no backpropagation.
However, I found that whatever the performance of pre-training model is, MMT can always achieve competitive performance.

from mmt.

I see.
Thanks for your reply!

from mmt.