Memory leak when saving model? about xnmt HOT 7 CLOSED

Yes, this should be easy enough to fix by passing the model around and adding a renew_cg(). I'll try to do it tonight.

from xnmt.

I am able to reproduce on OS X, but it goes away if I disable evaluating (which only occurs between epochs, so it's consistent with what you observe).

There's probably a memory leak in the decoder (I don't see how the evaluator could have one). Looking around, it seems never to call renew_cg(), so that could be the problem.

from xnmt.

Aah OK that's definitely it then

from xnmt.

Actually the way it currently works is that xnmt_train periodically writes the model to a file, and every time xnmt_decode needs to do decoding, it creates a new model and loads the latest parameters from that file. The issue could be that xnmt_decode's models are not properly garbage-collected.

renew_cg() isn't called by the decoder but it's called all the time by the trainer so that shouldn't be an issue, right?

from xnmt.

Ah yes. No renew_cg should only be called once per training/testing batch so it should be fine.

So the way to go should be to keep the same model at all time, which should be doable if there is no need to have a duplicate model at any time.

Or at least just create a training and testing model but keep them along the execution, you can't rely on garbage collection (for models and parameters) this is a low level dynet bug

from xnmt.

Hmm, could be related to this: clab/dynet#418
We should fix this...

But regardless, the decoder should call renw_cg().

from xnmt.

Hi!
I mem-topped the program calling renew_cg every time decoder is called and also the case where renew_cg is called for every translation, we might ideally call renew_cg() for every batch size of translations maybe. The decoder does sentence wise translation, so there is no concept of batch there. So the graph was technically cleared only after all the translations were done. This led to the memory leak. Sorry about that.

from xnmt.