Comments (5)
I have just checked in the xlnet model, which uses unigram lm segmentation based on sentence piece library. Our implementation already has a good parity and good performance but this is still work in progress. We plan to publish it by the end of April.
from blingfire.
It is done! We have published a new python package v 0.1.1 which has XLNET support in two ways:
- xlnet.bin has relevant NFKC normalization and segmentation
- xlnet_nonorm.bin does not have NFKC normalization (it assumes the text is already normalized) and only has segmentation
Both models have ~99.9% parity on input level comparing with the sentence piece library implementation, see corresponding readme.txt files in the BlingFire/ldbsrc/xlnet and BlingFire/ldbsrc/xlnet_nonorm directories.
from blingfire.
Please use this manual to create support for XLNet: https://github.com/Microsoft/BlingFire/wiki/How-to-add-a-new-BERT-tokenizer-model , I will be happy to help.
from blingfire.
@SergeiAlonichau
That would be an honor, but currently I have not enough knowledge in deep learning.
I plan to use tools like BlingFire after having studied NLP.
It would be better if someone more knowledgeable than me did this.
from blingfire.
Thanks @SergeiAlonichau! What do you think about tagging the latest release? This should make it easier to find/reference a specific release on GitHub.
from blingfire.
Related Issues (20)
- Byte offsets for original input bytes to allow non-destructive tokenization
- Trouble installing for custom model creation HOT 2
- M2M100 Marianmt tokenizers
- what is the last char of the last word from GetWords?
- How to create i2w model HOT 1
- Missing numpy dependency on setup.py
- Missing vcruntime140.dll and vcruntime140_1.dll dependencies HOT 4
- Could java call the tokenizer of bin file
- Add xlm-roberta-large tokenization support
- BlingFire fails with all-lowercase text
- "terminate called after throwing an instance of 'std::runtime_error'" HOT 1
- Issues building on Mac OSX M2 HOT 1
- Unable to Modify Tokenization Logic
- /O2 in CMakeLists.txt is incompatible with vcpkg using Ninja
- Import issue on MacOS M1 HOT 5
- Support for CLIP tokenizers from Hugging Face
- Build_Dll_For_Linux_ARM64 job fails
- c# example negative offset for Starts
- Loading the bert_base_tok.bin model sometimes throws an exception
- Unable to load DLL 'blingfiretokdll' or one of its dependencies: 找不到指定的模块。 (0x8007007E) System.DllNotFoundException: Unable to load DLL 'blingfiretokdll' or one of its dependencies: 找不到指定的模块。 (0x8007007E) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from blingfire.