Comments (4)
This deduplicator doesn't know anything about documents. It just knows strings. Do you have a document separator that you use that's not present in any of the documents? (e.g., if you have a tokenizer with <65k tokens you can use \xff\xff\xff\xff as a separator.
from deduplicate-text-datasets.
This deduplicator doesn't know anything about documents. It just knows strings. Do you have a document separator that you use that's not present in any of the documents? (e.g., if you have a tokenizer with <65k tokens you can use \xff\xff\xff\xff as a separator.该重复数据删除器对文档一无所知。它只知道字符串。您使用的文档分隔符是否存在于任何文档中? (例如,如果您的分词器具有 <65k 标记,则可以使用 \xff\xff\xff\xff 作为分隔符。
i use the \xff\xff as a separator . the tokenizer is gpt2 with <51k. Is there a big difference between "\xff\xff" and "\xff\xff\xff\xff"? thx for reply
from deduplicate-text-datasets.
Huh. If you can be sure that 0xff00 isn't a valid token then \xff\xff should work because you should never be able get away with 2. Do you put a unique counter between documents as well?
Otherwise it could match [final bit of document 1][document separator][beginning of document 2] to a document 3/4 if those were in the same positions.
from deduplicate-text-datasets.
Huh. If you can be sure that 0xff00 isn't a valid token then \xff\xff should work because you should never be able get away with 2. Do you put a unique counter between documents as well?呵呵。如果您可以确定 0xff00 不是有效令牌,那么 \xff\xff 应该可以工作,因为您永远无法逃脱 2. 您是否也在文档之间放置了唯一的计数器?
Otherwise it could match [final bit of document 1][document separator][beginning of document 2] to a document 3/4 if those were in the same positions.否则,它可以将[文档 1 的最后一位][文档分隔符][文档 2 的开头]与文档 3/4 匹配(如果它们位于相同的位置)。
maybe,I'll try it. thx
from deduplicate-text-datasets.
Related Issues (20)
- How to restore the result data after deduplication (remove invisible characters) HOT 1
- cargo build error.Could you upload cargo.lock file? HOT 2
- what is the input dataset format for custom dataset? HOT 1
- Retain one instance per duplicate HOT 2
- Adjust TensorFlow version to fix cuDNN, cuFFT, cuBLAS errors. HOT 4
- Incomplete Sentences HOT 1
- remove_ex in finish_dedup_wiki40b HOT 1
- [Question] An error with the same repo guideline HOT 5
- customized dataset deduplication HOT 1
- where the data is? HOT 1
- question about wstring_equal function HOT 2
- 是否可以提供一个纯python版本的,相信很多研究者在服务器上没有权限安装gcc HOT 1
- Count_occurrence does not work with tokenizer? HOT 2
- Question: Upper Bound HOT 1
- [Bug] Out of range error when counting occurrences on a custom suffix array HOT 1
- does this tool can process Chinese? HOT 1
- does finish_dedup_wiki40b.py has some wrong?
- Distributed running
- called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" } HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deduplicate-text-datasets.