robertostling / hnmt Goto Github PK

Implement coverage in the attention mechanism, following [1].

[1] Tu, Zhaopeng, et al. "Coverage-based Neural Machine Translation." arXiv preprint arXiv:1601.04811 (2016).
http://arxiv.org/pdf/1601.04811

Implement an encoder following [1].

[1] Lee, Jason, Kyunghyun Cho, and Thomas Hofmann. "Fully Character-Level Neural Machine Translation without Explicit Segmentation." arXiv preprint arXiv:1610.03017 (2016).
https://arxiv.org/pdf/1610.03017

I would be very appreciate if you perfect the documentation.

Vary the number of sentences per minibatch, so that the number of tokens (including padding) is close to optimal. The current fixed-size minibatches under-utilize the available memory when the minibatch is filled with short sentences.

Averaging parameters for a single training run improves results, as does ensembling several independently trained models. We should make the decoder able to combine these two methods, perhaps it's possible to squeeze out an extra bit of performance this way. It should be quick to implement.

Currently the whole training data set is loaded to RAM, but this obviously does not scale. At some point we need to fix this, if we want to train with huge corpora.

Pretrained models would be much appreciated. Do you provide them?

If not, then you could maybe add a statement about this in the readme?

Thanks!

Implement Minimum Risk Training (MRT), following [1].

[1] Shen, Shiqi, et al. "Minimum risk training for neural machine translation." arXiv preprint arXiv:1512.02433 (2015).
http://arxiv.org/pdf/1512.02433

Implement a two-level decoder following [1].

First a word-level decoder produces a sequence that may contain <UNK> symbols.
Next, these <UNK> symbols are filled in using a character-level decoder.

[1] Luong, Minh-Thang, and Christopher D. Manning. "Achieving open vocabulary neural machine translation with hybrid word-character models." arXiv preprint arXiv:1604.00788 (2016).
http://arxiv.org/pdf/1604.00788

ERROR (theano.gof.opt): SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x7f6bb55b4198>
ERROR (theano.gof.opt): Traceback:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/home/lnmiit/.local/lib/python3.6/site-packages/Theano-0.9.0-py3.6.egg/theano/gof/opt.py", line 235, in apply
sub_prof = optimizer.optimize(fgraph)
File "/home/lnmiit/.local/lib/python3.6/site-packages/Theano-0.9.0-py3.6.egg/theano/gof/opt.py", line 87, in optimize
ret = self.apply(fgraph, *args, **kwargs)
File "/home/lnmiit/.local/lib/python3.6/site-packages/Theano-0.9.0-py3.6.egg/theano/scan_module/scan_opt.py", line 685, in apply
node = self.process_node(fgraph, node)
File "/home/lnmiit/.local/lib/python3.6/site-packages/Theano-0.9.0-py3.6.egg/theano/scan_module/scan_opt.py", line 745, in process_node
node, args)
File "/home/lnmiit/.local/lib/python3.6/site-packages/Theano-0.9.0-py3.6.egg/theano/scan_module/scan_opt.py", line 854, in push_out_inner_vars
add_as_nitsots)
File "/home/lnmiit/.local/lib/python3.6/site-packages/Theano-0.9.0-py3.6.egg/theano/scan_module/scan_opt.py", line 906, in add_nitsot_outputs
reason='scanOp_pushout_output')
File "/home/lnmiit/.local/lib/python3.6/site-packages/Theano-0.9.0-py3.6.egg/theano/gof/toolbox.py", line 391, in replace_all_validate_remove
chk = fgraph.replace_all_validate(replacements, reason)
File "/home/lnmiit/.local/lib/python3.6/site-packages/Theano-0.9.0-py3.6.egg/theano/gof/toolbox.py", line 365, in replace_all_validate
fgraph.validate()
File "/home/lnmiit/.local/lib/python3.6/site-packages/Theano-0.9.0-py3.6.egg/theano/gof/toolbox.py", line 256, in validate_
ret = fgraph.execute_callbacks('validate')
File "/home/lnmiit/.local/lib/python3.6/site-packages/Theano-0.9.0-py3.6.egg/theano/gof/fg.py", line 589, in execute_callbacks
fn(self, *args, **kwargs)
File "/home/lnmiit/.local/lib/python3.6/site-packages/Theano-0.9.0-py3.6.egg/theano/gof/toolbox.py", line 422, in validate
raise theano.gof.InconsistencyError("Trying to reintroduce a removed node")
theano.gof.fg.InconsistencyError: Trying to reintroduce a removed node

Traceback (most recent call last):
File "hnmt.py", line 1426, in
if name == 'main': main()
File "hnmt.py", line 1214, in main
'Heldout training sentences is no longer supported')

Error received when trying to train the Machine Translation System using command

python3 hnmt.py --train europarl-v7.sv-en \

            --source-tokenizer word \
            --target-tokenizer char \
            --load-source-vocabulary vocab.sv \
            --load-target-vocabulary vocab.en \
            --batch-budget 32 \

--training-time 8
--log en-sv.log
--save-model en-sv.model