Inconsistency in CPU results and GPU results in model training about tensorflow HOT 4 CLOSED

Hi @Liqi1003 ,

The difference seems precision related because of XLA fusion. Please find the developer comment on same for more details.

from tensorflow.

Hi @SuryanarayanaY,

Thanks for the pointer!

To test whether it is a problem caused by XLA, I tried to disable XLA by adding TF_XLA_FLAGS=--tf_xla_auto_jit=-1 before the command, as mentioned here. However, I still see the log indicates XLA compilation is enabled, and the outputs are the same as the one I posted above. Am I using the wrong way to disable XLA?

Also, I was only able to reproduce this problem in tensorflow 2.13. Using google colab with tensorflow 2.16.1, running the same code does not result in such inconsistency. Here is the colab.

I wonder if it is a bug that was silently fixed in later versions? Thanks!

from tensorflow.

Hi @Liqi1003 ,

It seems 2.16v has better precision than earlier versions. May be there seems some internal amendments which I am not aware.
As it resolved in latest version can we mark this as closed. Thanks!

from tensorflow.

Are you satisfied with the resolution of your issue?
Yes
No

from tensorflow.