layer normalization in layer? about qanet HOT 3 OPEN

I tried this, but the error message show below：
ValueError: Dimensions must be equal, but are 128 and 32 for 'Embedding_Encoder_Layer/Encoder_Residual_Block/encoder_block_0/layer_norm_0/sub' (op: 'Sub') with input shapes: [32,?,1,128], [32].
I guess there are many places have to adjust to fit it XD

from qanet.

@alznn maybe you can try tf.nn.moments(inputs, axes). I use this, and no error. And you need to pay attention that after compute the mean and variance, you need to use the beta and gamma to renormalize the distribution, the shape of beta and gamma is inputs.shape[-1].

from qanet.

Thanks for your question! You are right in that right now it looks more like an instance normalization rather than a layer normalization. If you check this line I actually attempted using both normalization methods and they didn't show much difference so I sticked with one!

from qanet.