Code for the EMNLP 2021 Paper AfroMT: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languages.
@inproceedings{reid21afromt,
title = {Afro{MT}: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languages},
author = {Machel Reid and Junjie Hu and Graham Neubig and Yutaka Matsuo},
booktitle = {Conference on Empirical Methods in Natural Language Processing (EMNLP)},
address = {Punta Cana, Dominican Republic},
month = {November},
url = {https://arxiv.org/abs/2109.04715},
year = {2021}
}git clone [email protected]:machelreid/afromtPlease use gdown to download the data
cd afromt
gdown --id 1Qj3IXQ9kusaeHtGYRVPzMnGm_dQvvfJP
tar -xf afromt.tar.xzRun the following commands
cd training
# install apex for fp16 for faster training
git clone https://github.com/NVIDIA/apex
cd apex
pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
cd ../
# install fairseq
pip install -e .
cd ../Download the model as follows:
gdown --id 1A9kbWHnMrjFwgq9x8rpnU4aATx_Y4b3b
tar -xf afrobart.tar.xz # and you will get afrobart.pt which is the model fileFirst be sure to install sentencepiece
TGT_LANG=#whatever target language you choose
cd afromt # the data folder
cd en-$TGT_LANG
bash ../../preprocess.sh $TGT_LANG ../../
cd ../../to preprocess $TGT_LANG (e.g. zu for Zulu, or xh for Xhosa). This command will produce a data-bin-afromt folder in the folder for the en-$TGT_LANG language pair.
More details within the script itself if you want to tweak training arguments!
bash train_afrobart.sh afromt/en-$TGT_LANG/data-bin-afromt afromt/en-$TGT_LANG/model_output $TGT_LANGand you should see your model training!
Once training is over, you can generate from the model as follows:
fairseq-generate afromt/en-$TGT_LANG/data-bin-afromt --gen-subset test --path afromt/en-$TGT_LANG/model_outputs/checkpoints/checkpoint_best.pt --beam 5 --batch-size 300 --remove-bpe sentencepiece --truncate-source --task translation_from_xbart -s en -t $TGT_LANG --lenpen $LENPEN > generation.log (where $TGT_LANG and $LENPEN are variables controlling target language and length penalty respectively)
and generate BLEU scores as follows:
bash eval_bleu_chrf.sh generation.log- The bilingual dictionaries we extracted are in the
dictionaries/folder (note that these are automatically extracted with a word aligned so they are not gold standard)
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent