Nondeterministic behaviour in SpatialMaxPooling about cunn HOT 8 CLOSED

This is expected. Its Extremely hard to write a max pooling kernel that is deterministic when kW > dW or kH > dH. cunn, cudnn, caffe, all of these are non-deterministic maxpoolings for that case.

when kW <= dW, the kernel can be made deterministic with some effort (I think it is deterministic, but I haven't verified).
The atomic adds were done at some point for just handling the case of kW != dW, for correctness, and they are needed. Otherwise two threads might race.

from cunn.

Looking at the code in Caffe (https://github.com/BVLC/caffe/blob/master/src/caffe/layers/pooling_layer.cu), I believe their backward passes are deterministic: for each bottom element, all possible top elements that could contribute to it are scanned in a single thread, the summing order thus being deterministic. As another benefit, they don't need atomics.

from cunn.

So after empirical testing, I can say that all other implementations (caffe, cudnn V2, convnet2) behave deterministically. Therefore, I would like to reopen the issue. If this is not going to be fixed, it should be at least documented, similarly to the "Reproducibility" chapter in cudnn user guide. It would be also interesting to document how the convolution modules behave.

from cunn.

@wickedfoo investigated this, and this is not what he found. He found that cudnn's max pooling is clearly non-deterministic. @wickedfoo comments?

from cunn.

The behavior of cudnn's maxpool seems to have changed: while experimentally non-deterministic in V2R2, it appears to be ok in V2.

from cunn.

oh I see. Good to know that. The last investigation was indeed probably done in V1 or V2Rxx

from cunn.

I don't put my hand in fire for that, though:).
UPDATE: Actually, now I believe that cudnn.SpatialMaxPooling is not deterministic in V2 either. Sorry for confusion. But Caffe's maxpooling definitely is.

from cunn.

I think this has been addressed in dc77610

from cunn.