PyTorch implementations of seq2seq models for Neural Machine Translation (NMT) task:

• seq2seq (RNN)
• seq2seq with attention (RNN + attention)
• ConvS2S
• Transformer
• DynamicConv (+ LightConv)

Please refer to no-torchtext tag. In this version, the dataset.py, lang.py and data_prepare.py structuralize low-level text to make it easier to use in the training code.

Supporting datasets include pytorch tutorial ENG to FRA translation dataset and torchtext NMT datasets.

• org: ENG to FRA translation from pytorch tutorial
  • To use this data, please download dataset from https://download.pytorch.org/tutorial/data.zip first.
• multi30k
• iwslt
• wmt14

• python3
• pyyaml
• pytorch >= 1.10
• tensorboard >= 1.14
• torchtext
• spacy
  • python -m spacy download en
  • python -m spacy download de

Hparams:

• Task & data: ENG to FRA translation task, max_len=14, min_freq=2.

Models:

• [!] BLEU is recorded in different run
• PPL and BLEU does not match
• about the Transformer
  • after-norm does not work; should use before-norm.
  • LR warmup and xavier init is important for the performance

• Beam search
• Word tokenization
  • BPE
  • Word piece model

• https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html
• https://github.com/bentrevett/pytorch-seq2seq
• https://github.com/pytorch/fairseq