Add support for XLnet about blingfire HOT 5 CLOSED

I have just checked in the xlnet model, which uses unigram lm segmentation based on sentence piece library. Our implementation already has a good parity and good performance but this is still work in progress. We plan to publish it by the end of April.

from blingfire.

It is done! We have published a new python package v 0.1.1 which has XLNET support in two ways:

1. xlnet.bin has relevant NFKC normalization and segmentation
2. xlnet_nonorm.bin does not have NFKC normalization (it assumes the text is already normalized) and only has segmentation

Both models have ~99.9% parity on input level comparing with the sentence piece library implementation, see corresponding readme.txt files in the BlingFire/ldbsrc/xlnet and BlingFire/ldbsrc/xlnet_nonorm directories.

from blingfire.

Please use this manual to create support for XLNet: https://github.com/Microsoft/BlingFire/wiki/How-to-add-a-new-BERT-tokenizer-model , I will be happy to help.

from blingfire.

@SergeiAlonichau
That would be an honor, but currently I have not enough knowledge in deep learning.
I plan to use tools like BlingFire after having studied NLP.
It would be better if someone more knowledgeable than me did this.

from blingfire.

Thanks @SergeiAlonichau! What do you think about tagging the latest release? This should make it easier to find/reference a specific release on GitHub.

from blingfire.