demelin / nematus Goto Github PK

Attention-based encoder-decoder model for neural machine translation

This package is based on the dl4mt-tutorial by Kyunghyun Cho et al. ( https://github.com/nyu-dl/dl4mt-tutorial ). It was used to produce top-scoring systems at the WMT 16 shared translation task.

The changes to Nematus include:

• new architecture variants for better performance:

  • arbitrary input features (factored neural machine translation) http://www.statmt.org/wmt16/pdf/W16-2209.pdf
  • deep models (Zhou et al., 2016; Wu et al., 2016; Miceli Barone et al., 2017) https://arxiv.org/abs/1606.04199 https://arxiv.org/abs/1609.08144 https://arxiv.org/abs/1707.07631
  • dropout on all layers (Gal, 2015) http://arxiv.org/abs/1512.05287
  • tied embeddings (Press and Wolf, 2016) https://arxiv.org/abs/1608.05859
  • layer normalisation (Ba et al, 2016) https://arxiv.org/abs/1607.06450
  • weight normalisation (Salimans and Kingma, 2016) https://arxiv.org/abs/1602.07868

• improvements to scoring and decoding:

  • ensemble decoding (and new translation API to support it)
  • n-best output for decoder
  • scripts for scoring (given parallel corpus) and rescoring (of n-best output)

• usability improvements:

  • command line interface for training
  • more output options (attention weights; word-level probabilities) and visualization scripts
  • execute arbitrary validation scripts (for BLEU early stopping)
  • vocabulary files and model parameters are stored in JSON format (backward-compatible loading)
  • server mode

see changelog for more info

For general support requests, there is a Google Groups mailing list at https://groups.google.com/d/forum/nematus-support . You can also send an e-mail to [email protected] .

Nematus requires the following packages:

• Python 2.7
• numpy
• Theano >= 0.7 (and its dependencies).

we recommend executing the following command in a Python virtual environment: pip install numpy numexpr cython tables theano bottle bottle-log tornado

the following packages are optional, but highly recommended

• CUDA >= 7 (only GPU training is sufficiently fast)
• cuDNN >= 4 (speeds up training substantially)

you can run Nematus locally. To install it, execute python setup.py install

You can also create docker image by running following command, where you change suffix to either cpu or gpu:

docker build -t nematus-docker -f Dockerfile.suffix .

To run a CPU docker instance with the current working directory shared with the Docker container, execute:

docker run -v `pwd`:/playground -it nematus-docker

For GPU you need to have nvidia-docker installed and run:

nvidia-docker run -v `pwd`:/playground -it nematus-docker

Training speed depends heavily on having appropriate hardware (ideally a recent NVIDIA GPU), and having installed the appropriate software packages.

To test your setup, we provide some speed benchmarks with `test/test_train.sh', on an Intel Xeon CPU E5-2620 v3, with a Nvidia GeForce GTX 1080 GPU and CUDA 8.0:

CPU, theano 0.8.2:

THEANO_FLAGS=mode=FAST_RUN,floatX=float32,device=cpu ./test_train.sh

2.37 sentences/s

GPU, no CuDNN, theano 0.8.2:

THEANO_FLAGS=mode=FAST_RUN,floatX=float32,device=gpu ./test_train.sh

71.62 sentences/s

GPU, CuDNN 5.1, theano 0.8.2:

THEANO_FLAGS=mode=FAST_RUN,floatX=float32,device=gpu ./test_train.sh

139.73 sentences/s

GPU, CuDNN 5.1, theano 0.9.0dev5.dev-d5520e:

THEANO_FLAGS=mode=FAST_RUN,floatX=float32,device=gpu ./test_train.sh

173.15 sentences/s

GPU, CuDNN 5.1, theano 0.9.0dev5.dev-d5520e, new GPU backend:

THEANO_FLAGS=mode=FAST_RUN,floatX=float32,device=cuda ./test_train.sh

209.21 sentences/s

GPU, float16, CuDNN 5.1, theano 0.9.0-RELEASE, new GPU backend:

222.28 sentences/s

See SPEED.md for more benchmark results on different hardware and hyperparameter configurations.

execute nematus/nmt.py to train a model.

more instructions to train a model, including a sample configuration and preprocessing scripts, are provided in https://github.com/rsennrich/wmt16-scripts

The n-best list is assumed to have the same format as Moses:

sentence-ID (starting from 0) ||| translation ||| scores

new scores will be appended to the end. rescore.py has the same arguments as score.py, with the exception of this additional parameter:

sample models, and instructions on using them for translation, are provided in the test directory, and at http://statmt.org/rsennrich/wmt16_systems/

• The pipe symbol "|" serves as a factor separator and should not otherwise appear in the text. See the documentation on factors for more information. Using the Moses tokenizer also escapes pipe symbols.

• Support for float16 may not be fully functional or efficient using depending on the Theano version and GPU model. If you use float16 for training, consider using a lower learning rate for increased numerical stability.

if you use Nematus, please cite the following paper:

Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry and Maria Nadejde (2017): Nematus: a Toolkit for Neural Machine Translation. In Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Valencia, Spain, pp. 65-68.

@InProceedings{sennrich-EtAl:2017:EACLDemo,
  author    = {Sennrich, Rico  and  Firat, Orhan  and  Cho, Kyunghyun  and  Birch, Alexandra  and  Haddow, Barry  and  Hitschler, Julian  and  Junczys-Dowmunt, Marcin  and  L\"{a}ubli, Samuel  and  Miceli Barone, Antonio Valerio  and  Mokry, Jozef  and  Nadejde, Maria},
  title     = {Nematus: a Toolkit for Neural Machine Translation},
  booktitle = {Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics},
  month     = {April},
  year      = {2017},
  address   = {Valencia, Spain},
  publisher = {Association for Computational Linguistics},
  pages     = {65--68},
  url       = {http://aclweb.org/anthology/E17-3017}
}

the code is based on the following model:

Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio (2015): Neural Machine Translation by Jointly Learning to Align and Translate, Proceedings of the International Conference on Learning Representations (ICLR).

please refer to the Nematus paper for a description of implementation differences

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreements 645452 (QT21), 644333 (TraMOOC), 644402 (HimL) and 688139 (SUMMA).