Subsampling frequent outputs about xlm HOT 4 CLOSED

Hi,

Not sure about what you mean by frequent outputs, but the code that selects the words to mask is here: https://github.com/facebookresearch/XLM/blob/master/src/trainer.py#L295-L305

sample_alpha == 0 will do the same thing as in the original BERT paper, i.e. sample 15% of words at random. A non-zero value of sample_alpha will sample 15% of the words in a batch, but each word will have a different probability of being masked out (i.e. the rare words will have a higher probability to be masked out than the frequent ones).

from xlm.

Yes, I was referring to

we also subsample the frequent outputs using an approach similar to Mikolov et al. (2013b)

Thanks for pointing me to the function.

from xlm.

Sorry, one more question: which value for sample_alpha have you used in your experiments?

from xlm.

We used sample_alpha = 0.5

from xlm.