Count_occurrence does not work with tokenizer? about deduplicate-text-datasets HOT 2 CLOSED

Yeah. This project works in byte-space, not "word"-space. If you tokenize your dataset, then you'll have to handle things yourself if you want different tokens. (If your dataset isn't that big, you can also just work in byte-space and everything will be fine and easy.)

For your specific example, you'll want to make sure to add an initial space b = " Architecturally" and everything should match what you expect. This is because most tokenizers add the space to the following word. So it's not "that" context-dependent, just what the one prior byte was.

If you want to do this properly to do word searching, you'll want to find all first tokens that contain your word and start from there. I don't know exactly how "Architecturally" is tokenized but let's pretend it's "Arch" "itect" "urally". This means you'll want to find all tokens that contain "Arch" which might be "\nArch", " Arch", "Arch", "\tArch". Then run your search for each of those followed by the remaining 2 tokens.

from deduplicate-text-datasets.

Hi Nicholas,

Okay thank you for prompt reply! This helps a lot and indeed by adding a space at the start I was able to find most of the sequences in SQUAD.

Have a good day!

from deduplicate-text-datasets.