The BLEU decreased when train on Unsupervised NMT about xlm HOT 6 CLOSED

I'm surprised by the perplexity of your language model:

valid_mn_mlm_ppl -> 12.698742
valid_zh_mlm_ppl -> 482.045657

482 is really high, I'm afraid the model does not encode Chinese sentences properly here. Can you pretrain for a longer time? Did you stop at epoch 11 because the model had stopped converging?

Another thing to check is whether the word embeddings in the pretrained LM lookup table are somehow aligned. You can for instance print the nearest neighbors of Chinese words and see if they are close to their Mongolian translations.

from xlm.

How can I get the nearest neighbors of Chinese words？@glample

from xlm.

This notebook will show you how to reload a model with the associated dictionary: https://github.com/facebookresearch/XLM/blob/master/generate-embeddings.ipynb

Once you have it, you can simply extract the embeddings with model.embeddings. Then, for a given Chinese word X, just look for model.embeddings[dico.index(X)] it will give you the embedding of the word. You can then simply do some nearest neighbors search of the closest vectors in the embeddings, and map word ids to their original words with dico[word_id].

from xlm.

If the word embeddings in the pretrained LM lookup table are aligned not good，How can I improve it.@glample

from xlm.

First you would need to fix the language model quality on the Chinese sentences, the perplexity should be much lower than this, so right now it is not surprising that it does not work. Maybe train with more data / check that the Chinese segmentation is correct and that there is a not a bug in your data preprocessing? Or simply train longer / with more GPUs, as 11 epochs is not much.

from xlm.

I got it. Thanks！

from xlm.