Comments (7)
Yes, this should be easy enough to fix by passing the model around and adding a renew_cg(). I'll try to do it tonight.
from xnmt.
I am able to reproduce on OS X, but it goes away if I disable evaluating (which only occurs between epochs, so it's consistent with what you observe).
There's probably a memory leak in the decoder (I don't see how the evaluator could have one). Looking around, it seems never to call renew_cg()
, so that could be the problem.
from xnmt.
Aah OK that's definitely it then
from xnmt.
Actually the way it currently works is that xnmt_train periodically writes the model to a file, and every time xnmt_decode needs to do decoding, it creates a new model and loads the latest parameters from that file. The issue could be that xnmt_decode's models are not properly garbage-collected.
renew_cg()
isn't called by the decoder but it's called all the time by the trainer so that shouldn't be an issue, right?
from xnmt.
Ah yes. No renew_cg should only be called once per training/testing batch so it should be fine.
So the way to go should be to keep the same model at all time, which should be doable if there is no need to have a duplicate model at any time.
Or at least just create a training and testing model but keep them along the execution, you can't rely on garbage collection (for models and parameters) this is a low level dynet bug
from xnmt.
Hmm, could be related to this: clab/dynet#418
We should fix this...
But regardless, the decoder should call renw_cg().
from xnmt.
Hi!
I mem-topped the program calling renew_cg every time decoder is called and also the case where renew_cg is called for every translation, we might ideally call renew_cg() for every batch size of translations maybe. The decoder does sentence wise translation, so there is no concept of batch there. So the graph was technically cleared only after all the translations were done. This led to the memory leak. Sorry about that.
from xnmt.
Related Issues (20)
- Search Strategies assume decoder supports calc_log_probs HOT 2
- Overabundance of forced decoding
- Stative Attenders HOT 3
- Post Processes on Dev/Test References HOT 1
- Recipe las-pyramidal.yaml does not work HOT 13
- @register_xnmt_event problems
- Problem while using SAAMSeqTransducer HOT 6
- Confusing error message when YAML file doesn't exist
- Output silently fails when path contains ~
- How to load an existing trained model and make predictions on the unseen data? HOT 1
- dy.sigmoid doesn't exist HOT 1
- Pyyaml Error when running example 19, subword sample
- how can I use for character-level sequences?
- Running 16_transformer.yaml: AssertionError: cannot pass dy.ParameterCollection to a Serializable class. HOT 1
- Segment fatal (core dump) using 3.8.0 HOT 1
- Problem running GPU version on RTX 2070 with 8GB Memory HOT 1
- How to train XNMT with datasets that do not fit in RAM? HOT 1
- How to use already tokenized BPE text and BPE vocab with xnmt ?
- Error when running experiments
- Can't run test/config/transformer.yaml
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from xnmt.