Comments (3)
Hi,
The source code for sentence breaking is here: BlingFire/ldbsrc/sbd . The specific file is: wbd.lex.utf8 .
The documentation file describing the syntax of the rules is here: BlingFire/doc/lex.htm .
In order to have it recompiled please follow these steps: https://github.com/microsoft/BlingFire/wiki/How-to-change-linguistic-resources .
from blingfire.
Hi all, @SergeiAlonichau
I have some questions. I've followed the instruction from @SergeiAlonichau, modified the file sbd/wbd.lex.utf8
and recompiled using the command make -f Makefile.gnu lang=sbd all
.
Then, from dist-pypi
I rebuilt the python package, using python setup.py sdist
but I have the feeling my changes were not taken in consideration.
The file ldbsrc/ldb/sbd.bin
which is the output of the make command, doesn't seems to be included in the setup.py
.
What did I miss? I would like to have a new version of blingfilre packaged as a python package with the new segmentation rules.
Thank you in advance
I deleted and re-wrote this comment as I got more information
from blingfire.
Hi,
Thank you for your effort and sorry for the late reply!
The text_to_sentences uses built in model. The generated bin is converted to CPP array (see this https://raw.githubusercontent.com/microsoft/BlingFire/master/ldbsrc/sbd/BlingFireTokLibSbdData.cxx it has a command line in the comments) and that array is used instead of a model loaded from file.
However you can always load your own model if you use text_to_sentences_with_model. So that should be working. In fact I suggest to run two models side by side and compare differences this way.
If you wish you can also ship it this way, call it sbd2 or anything else you like.
If you want to replace the default model, we need to make sure it is better than the default model so some good amount of testing is required.
Once you are convinced it is better, just generate a CXX file from your binary and replace the existing *cxx file with yours. Then text_to_sentences will use your model.
I am ruling out that you did not install your local package correctly of course but maybe worth double checking.
from blingfire.
Related Issues (20)
- Byte offsets for original input bytes to allow non-destructive tokenization
- Trouble installing for custom model creation HOT 2
- M2M100 Marianmt tokenizers
- what is the last char of the last word from GetWords?
- How to create i2w model HOT 1
- Missing numpy dependency on setup.py
- Missing vcruntime140.dll and vcruntime140_1.dll dependencies HOT 4
- Could java call the tokenizer of bin file
- Add xlm-roberta-large tokenization support
- BlingFire fails with all-lowercase text
- "terminate called after throwing an instance of 'std::runtime_error'" HOT 1
- Issues building on Mac OSX M2 HOT 1
- Unable to Modify Tokenization Logic
- /O2 in CMakeLists.txt is incompatible with vcpkg using Ninja
- Import issue on MacOS M1 HOT 5
- Support for CLIP tokenizers from Hugging Face
- Build_Dll_For_Linux_ARM64 job fails
- c# example negative offset for Starts
- Loading the bert_base_tok.bin model sometimes throws an exception
- Unable to load DLL 'blingfiretokdll' or one of its dependencies: 找不到指定的模块。 (0x8007007E) System.DllNotFoundException: Unable to load DLL 'blingfiretokdll' or one of its dependencies: 找不到指定的模块。 (0x8007007E) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from blingfire.