Reddit_TFT Training error about cloudml-samples HOT 6 CLOSED

@AkhilSinghRana Can you please try this with TF 1.0 and let us know if it works.

from cloudml-samples.

@puneith thanks for your response. It seems to be working with tensorflow1.0 but there are still some warnings related to GPU and float32 variables. Will let you know when the task gets completed. Thanks again.

from cloudml-samples.

@puneith Also do you have any idea how long it takes for the training to complete on cloud ? Or how many iterations will it continue for ? There is no accuracy report in between will it be shown at last! after complete training is done ? It just shows the Loss and step values. Hope it is working fine

from cloudml-samples.

@AkhilSinghRana Number of steps is a specified value, check the defaults if you did not specify. Closing this for now.

from cloudml-samples.

I have the same problem with tft 0.1.10 and tf 1.3. After reading this issue tried to install tf 1.0, but it does not help me.

tensorflow.python.training.basic_session_run_hooks.NanLossDuringTrainingError: NaN loss during training.

from cloudml-samples.

@lukashes Could you provide more complete logs in a gist or something? For what it's worth these examples have not yet been updated to TF 1.3 as CMLE does not yet have a TF 1.3 runtime, but 1.3 should be backwards compat with 1.2. Still without seeing more detailed logs the only recommendation I can make is try TF 1.2.

from cloudml-samples.