When training for longer time (>7 hours) fails to save checkpoint about deep-makeover HOT 5 CLOSED

That is interesting. Some googling showed a potentially related issue: tensorflow/tensorflow#2654

Have you tried any of the ideas in there?

from deep-makeover.

OK, I think I know what is happening. Function _save_image() in dm_train.py is repeatedly adding new nodes to the graph.

While I fix that problem you could increase the value of summary_period so that this operation doesn't happen so frequently.

Let me know if that helps.

from deep-makeover.

OK, so I switched _save_image() to use numpy instead of tensorflow. I think that should largely take care of the problem of ever-growing checkpoints.

Could you please try commit c2ae55c..779100c and let me know whether that helps?

from deep-makeover.

Based on my testing the checkpoint size went down to ~20MB after 40K batches, so I'll be closing this issue.

Thanks for reporting it!

from deep-makeover.

I have also retested it after your update and it works fine now.

Thank you for your help!

from deep-makeover.