2080ti one gpu gives out of memory error while training about surfemb HOT 3 CLOSED

That's weird.
Disabling logging won't break anything, but if you want logging, try to update pytorch lightning, in case it's a bug in a specific version, or call .item() on the losses to be logged.

from surfemb.

Hi

It shouldn't use that much memory. I'm training on a single RTX 2080 with 8 GB of ram.
Also, I don't see why changing the number of scenes would affect GPU memory usage.
I'm not sure what's going on there.
Let me know, if you find out.

BR, Rasmus

from surfemb.

Hi, thanks for the quick response.

I have noticed that there is a memory leak on surface_embedding.py
I guess it is logging during training and thus it increases the memory usage in every 4-5 seconds. I have checked it with "nvidia-smi" command during training

So I have commented on these 3 lines and now it even works with batch-size=32 and multiple threads :)
self.log(f'{log_prefix}/loss', loss)
self.log(f'{log_prefix}/mask_loss', mask_loss)
self.log(f'{log_prefix}/nce_loss', nce_loss)

I hope this disabled logging wont break things

thanks again for the great work,

from surfemb.