when i use tokenizer , I obtained many patterns that span across the data, which is quite strange. about deduplicate-text-datasets HOT 4 OPEN

This deduplicator doesn't know anything about documents. It just knows strings. Do you have a document separator that you use that's not present in any of the documents? (e.g., if you have a tokenizer with <65k tokens you can use \xff\xff\xff\xff as a separator.

from deduplicate-text-datasets.

This deduplicator doesn't know anything about documents. It just knows strings. Do you have a document separator that you use that's not present in any of the documents? (e.g., if you have a tokenizer with <65k tokens you can use \xff\xff\xff\xff as a separator.该重复数据删除器对文档一无所知。它只知道字符串。您使用的文档分隔符是否存在于任何文档中？ （例如，如果您的分词器具有 <65k 标记，则可以使用 \xff\xff\xff\xff 作为分隔符。

i use the \xff\xff as a separator . the tokenizer is gpt2 with <51k. Is there a big difference between "\xff\xff" and "\xff\xff\xff\xff"? thx for reply

from deduplicate-text-datasets.

Huh. If you can be sure that 0xff00 isn't a valid token then \xff\xff should work because you should never be able get away with 2. Do you put a unique counter between documents as well?

Otherwise it could match [final bit of document 1][document separator][beginning of document 2] to a document 3/4 if those were in the same positions.

from deduplicate-text-datasets.

Huh. If you can be sure that 0xff00 isn't a valid token then \xff\xff should work because you should never be able get away with 2. Do you put a unique counter between documents as well?呵呵。如果您可以确定 0xff00 不是有效令牌，那么 \xff\xff 应该可以工作，因为您永远无法逃脱 2. 您是否也在文档之间放置了唯一的计数器？

Otherwise it could match [final bit of document 1][document separator][beginning of document 2] to a document 3/4 if those were in the same positions.否则，它可以将[文档 1 的最后一位][文档分隔符][文档 2 的开头]与文档 3/4 匹配（如果它们位于相同的位置）。

maybe，I'll try it. thx

from deduplicate-text-datasets.