Error with table size not being divisible by text size about deduplicate-text-datasets HOT 7 CLOSED

Huh. Thanks for reporting this. I'm not sure what's going on right now but I'll look into it later this week.

To help me debug, could you upload (assuming it's safe to do so, but because you're saying it's wikipedia data I suspect it is) the exact file you're using in case that makes a difference?

from deduplicate-text-datasets.

Yes absolutely! Here is the file test.txt (note: it's very small) I just copied the first few paragraphs of random articles (hint: they were Youtube, Google, and American Revolutionary War). Then I took line5 and line16 and duplicated them.

I ran this command bash scripts/deduplicate_single_file.sh test.txt test_dedup.txt 20 1 and got the following output:

./target/debug/dedup_dataset make-part --data-file test.txt --start-byte 0 --end-byte 10490
Waiting for jobs to finish
Checking all wrote correctly
FACT 2.0
Rerunning 0 jobs because they failed.
Merging suffix trees
./target/debug/dedup_dataset merge --output-file tmp/out.table.bin --suffix-path test.txt.part.0-10490 --num-threads 40
Now merging individual tables
Cleaning up
    Finished dev [optimized + debuginfo] target(s) in 0.06s
     Running `target/debug/dedup_dataset self-similar --data-file test.txt --length-threshold 20 --cache-dir /tmp/cache --num-threads 1`
Start load!
thread 'main' panicked at 'assertion failed: metadata.len() % (text.len() as u64) == 0', src/main.rs:479:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    Finished dev [optimized + debuginfo] target(s) in 0.03s
     Running `target/debug/dedup_dataset collect --data-file test.txt --cache-dir /tmp/cache --length-threshold 20`
thread 'main' panicked at 'assertion failed: size_table % size_text == 0', src/main.rs:1049:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

from deduplicate-text-datasets.

I also have an unrelated question: Can this program handle UTF-8 text (e.g. Korean)?

Second, we don't want UTF8 strings. Everything is a [u8] byte array, because we might be working over token sequences which aren't valid UTF8.

I wasn't sure what this meant in the README.md. If we were to pass a file with utf-8 strings as input, does this mean we are essentially ignoring the utf-8 encoding and compare each byte individually for substring match?

from deduplicate-text-datasets.

Yeah, you're right in how it handles things: it doesn't understand the idea of a "letter" or "token" or anything -- it just thinks in terms of "bytes".

from deduplicate-text-datasets.

Okay this is very weird. I can't reproduce the error you see: when I run the commands everything comes out as expected.

Could you re-run the command and upload the following files:

test.txt.part.0-10490
test.txt.part.0-10490.table.bin
test.txt.table.bin

from deduplicate-text-datasets.

I think I figured out what the issue was. When I cleared the tmp folder and ran the command again, the error went away. My guess is that b/c the tmp folder never got cleared, I was merging tables from previous runs too. Maybe we should have this line come after L91?

from deduplicate-text-datasets.

Ah -- perfect yes thanks. I will do that.

from deduplicate-text-datasets.