Comments (2)
from blingfire.
Unfortunately, TextToIds
doesn't seem to be an adequate work-around for TokenToId
. I am using the xlnet model and running TextToIds('<sep>')
returns [17, 4]
. Sure, I could just dump the first token and only respect the 4
, which is the correct answer. However, running TextToIds('<unk>')
returns [7739, 12287, 3151]
, which isn't even close to accurate since <unk>=0
. A proper TokenToId
function would be really useful. Are you open to accepting PRs?
from blingfire.
Related Issues (20)
- Byte offsets for original input bytes to allow non-destructive tokenization
- Trouble installing for custom model creation HOT 2
- M2M100 Marianmt tokenizers
- what is the last char of the last word from GetWords?
- How to create i2w model HOT 1
- Missing numpy dependency on setup.py
- Missing vcruntime140.dll and vcruntime140_1.dll dependencies HOT 4
- Could java call the tokenizer of bin file
- Add xlm-roberta-large tokenization support
- BlingFire fails with all-lowercase text
- "terminate called after throwing an instance of 'std::runtime_error'" HOT 1
- Issues building on Mac OSX M2 HOT 1
- Unable to Modify Tokenization Logic
- /O2 in CMakeLists.txt is incompatible with vcpkg using Ninja
- Import issue on MacOS M1 HOT 5
- Support for CLIP tokenizers from Hugging Face
- Build_Dll_For_Linux_ARM64 job fails
- c# example negative offset for Starts
- Loading the bert_base_tok.bin model sometimes throws an exception
- Unable to load DLL 'blingfiretokdll' or one of its dependencies: 找不到指定的模块。 (0x8007007E) System.DllNotFoundException: Unable to load DLL 'blingfiretokdll' or one of its dependencies: 找不到指定的模块。 (0x8007007E) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from blingfire.