Memory about holy-edge HOT 10 OPEN

my GPU only has 6G memory which is fine to run the code. dont know your setting

from holy-edge.

Sorry,I forgot to tell you.I use 1080ti GPU and 32GB memory.I can run your code with my own data.But Memory error is happened when the iteration is about 13000.I obverse top command in ubuntu, i find Memory is insufficient（not gpu memory, is computer memory).If you could tell me some suggestion,I really appreciate what you’ve done。

from holy-edge.

Sorry, I have not had such error. This code will save a checkpoint each 100 steps, maybe 13000 is too big in your dataset( I trained at 30000 steps without memory error).

from holy-edge.

I already changed to save a checkpoint with each 1000 steps, but the error is still happen.Did you run this code on windows? Thank you.

from holy-edge.

I run it on Ubuntu 16.04.

from holy-edge.

Could you please give details of your data-set ? It might be the case that your image sizes are quite large and since the VGG base model is full convolutional the intermediate representations overflow they GPU memory.

from holy-edge.

I also encountered the same problem with you, my server configuration is 8 Tesla p20, memory is 512G. Even with this configuration, memory error occurs after approximately 13,000 iterations during training. Will you solve the problem?@CangHaiQingYue

from holy-edge.

@Jasontachiangwu Well, when I cancelled the 'summary_write' operator, this problem was gone.
So I guess there maybe some bug in 'summary'. You may rebuild your own code.

from holy-edge.

@CangHaiQingYue，After I updated the tensorflow version to r1.8, no problem was found after training. It may be that there is a bug in the summary.

from holy-edge.

with same code on windows 10 and ubuntu 16.04, tensorflow v1.4.0, 20000 iterations:
memory error occurs in ubuntu 16.04
while no error occurs in windows 10

from holy-edge.