Data normalization about mcfly HOT 2 CLOSED

CNN and RNN start with batch normalization now. For the recurrent models there might be something better coming called Layer Normalization but the method only just got published (See Ba et al. 2016 in literature drive) and the layer is not yet in Keras.

from mcfly.

The literature all mentions that batch normalization is inserted after affine, before activation. The paper of Ioffe and Szegedy, 2015 (where batch norm is introduced) mentions:

We add the BN transform immediately before the
nonlinearity, by normalizing x = Wu+ b. We could have
also normalized the layer inputs u, but since u is likely
the output of another nonlinearity, the shape of its distribution
is likely to change during training, and constraining
its first and second moments would not eliminate the covariate
shift. In contrast, Wu + b is more likely to have
a symmetric, non-sparse distribution, that is “more Gaussian”
(Hyv¨arinen & Oja, 2000); normalizing it is likely to
produce activations with a stable distribution

So maybe we should not add it after the input layer, but after the first affine/convolutional layer.
It's not clear to me whether we should still also normalize the input. or that that's not necessary anymore.

from mcfly.