Freezing instance normalization during test phase? about cyclegan HOT 3 OPEN

From http://pytorch.org/docs/nn.html:

At evaluation time (.eval()), the default behaviour of the InstanceNorm module stays the same i.e. running mean/variance is NOT used for normalization. One can force using stored mean and variance with .train(False) method.

However, the results might not look so good since instance norm isn't really meant to be used this way -- this will induce a big gap between how you trained instance norm and how it is being used at test time.

If it doesn't look good, you could try retraining in a way that removes this gap. One way would be "virtual instance norm", where you normalize one image based on the running statistics from a held out set of other images (same as virtual batchnorm from this paper).

Good luck and I'll be interested to know if you have success!

from cyclegan.

Hi, torch implementation of Instance Norm is easily hackable and can use stored mean/var instead of independent normalization at test time. I think you just need to set self.bn to eval mode in https://github.com/junyanz/CycleGAN/blob/master/util/InstanceNormalization.lua#L102 or somewhere in updateOutput function. Stored statistics worked quite well for stylization.

Ideally, you want to get statistics from an entire image, you can create a dataset of tiles and compute global statistcs. I saw a script by @szagoruyko which does it, but could not find it now.

from cyclegan.

@DmitryUlyanov Thanks for the notes.

@Quasimondo I guess the tiling operation will still cause the boundary artifacts even without instanceNorm.

Let's say your images are 512x512. I would recommend that you train your model on 256x256 cropped images. During the test time, you are still able to test it on large images (512x512), as testing requires less GPU memory, compared to training. In this case, you might want to set loadSize=512 and fineSize=256 for both training and test. Even you train a model on 256x256 images, you can still apply it fully convolutionally to a picture with arbitrary size.

See issue #19 for more discussion.

from cyclegan.