Memory leak about unflow HOT 8 CLOSED

Hi, are you training on CPU or GPU? How long did you train? Can you give more details of your environment (os, tensorflow version, ...). I trained for multiple days on different machines and it didn't crash due to a leak.

from unflow.

Training with GPU, GTX Titan 1080. Ubuntu 16.04, tensorflow 1.2. During training, the GPU memory didn't explode, but the CPU memory increase continuously until 100%

from unflow.

I noticed that if I set the " tf.app.flags.DEFINE_boolean('debug', True)", the memory consumption will increase fast.

from unflow.

This flag enables recording and writing images to tensorboard, but i don't really know why it should cause a memory leak. On what timescale does the memory increase?

from unflow.

50000 iteration, it will consume near 30G memory

from unflow.

Is the 30GB with or without the debug Flag? The flag isn't meant to be used during full training. Could you maybe share your complete config.ini file with me?

from unflow.

The 30GB with the debug Flag, this is the config.ini.
config.ini.txt

from unflow.

As I was not able to reproduce, I am closing this for now.

from unflow.