Question of function run_balanced about visda2019 HOT 4 CLOSED

Hi @Antinomy20001,

Yes this function is a little bit tricky. The thing is that during one training step we need to pass 1 source and 2 target batches throw the same model. The sum of these 3 batch sizes doesn't allow us to pass them throw the model simultaneously due to the GPU memory limit. So we call the model 3 times. But here we are going to have troubles with BatchNorm statistics as they are not the same for source and target domains. This makes training process slower and much more unstable. The solution here is to form 3 batches with 1/3 of source images and 2/3 of target images in each of them.

Also, as I remember it is not our novelty, and we follow one of the open sourced pytorch implementations.

from visda2019.

Hi @Antinomy20001,

Yes this function is a little bit tricky. The thing is that during one training step we need to pass 1 source and 2 target batches throw the same model. The sum of these 3 batch sizes doesn't allow us to pass them throw the model simultaneously due to the GPU memory limit. So we call the model 3 times. But here we are going to have troubles with BatchNorm statistics as they are not the same for source and target domains. This makes training process slower and much more unstable. The solution here is to form 3 batches with 1/3 of source images and 2/3 of target images in each of them.

Also, as I remember it is not our novelty, and we follow one of the open sourced pytorch implementations.

Thanks for your rapid response! In fact, I am migrating your code to Pytorch, hahaha...😜.

Is this transpose operation to make BatchNorm's running_mean and running_var more stable? So call the function balanced? I think I have got it.

from visda2019.

Yes, you are right.

from visda2019.

Thanks again for your response and help

from visda2019.