Comments (2)
Yeah. This project works in byte-space, not "word"-space. If you tokenize your dataset, then you'll have to handle things yourself if you want different tokens. (If your dataset isn't that big, you can also just work in byte-space and everything will be fine and easy.)
For your specific example, you'll want to make sure to add an initial space b = " Architecturally"
and everything should match what you expect. This is because most tokenizers add the space to the following word. So it's not "that" context-dependent, just what the one prior byte was.
If you want to do this properly to do word searching, you'll want to find all first tokens that contain your word and start from there. I don't know exactly how "Architecturally" is tokenized but let's pretend it's "Arch" "itect" "urally". This means you'll want to find all tokens that contain "Arch" which might be "\nArch", " Arch", "Arch", "\tArch". Then run your search for each of those followed by the remaining 2 tokens.
from deduplicate-text-datasets.
Hi Nicholas,
Okay thank you for prompt reply! This helps a lot and indeed by adding a space at the start I was able to find most of the sequences in SQUAD.
Have a good day!
from deduplicate-text-datasets.
Related Issues (20)
- How to restore the result data after deduplication (remove invisible characters) HOT 1
- cargo build error.Could you upload cargo.lock file? HOT 2
- what is the input dataset format for custom dataset? HOT 1
- Retain one instance per duplicate HOT 2
- Adjust TensorFlow version to fix cuDNN, cuFFT, cuBLAS errors. HOT 4
- Incomplete Sentences HOT 1
- remove_ex in finish_dedup_wiki40b HOT 1
- [Question] An error with the same repo guideline HOT 5
- customized dataset deduplication HOT 1
- when i use tokenizer , I obtained many patterns that span across the data, which is quite strange. HOT 4
- where the data is? HOT 1
- question about wstring_equal function HOT 2
- 是否可以提供一个纯python版本的,相信很多研究者在服务器上没有权限安装gcc HOT 1
- Question: Upper Bound HOT 1
- [Bug] Out of range error when counting occurrences on a custom suffix array HOT 1
- does this tool can process Chinese? HOT 1
- does finish_dedup_wiki40b.py has some wrong?
- Distributed running
- called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" } HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deduplicate-text-datasets.