How much memory does it use? about dedup HOT 3 OPEN

The maximum amount of memory used for mmap mode depends on the number of distinct entries in the input. The program itself only uses 24 bytes of memory for each distinct entry but the peak run time usage will be significantly (2x-8x) higher than that, due to two copies of the internal hash table temporarily existing during reallocations. On the enwiki9 benchmark (1 GB dataset), the master branch hits 600MB peak memory usage for me (the data set just barely triggers a reallocation to the largest size).

I've been planning on implementing a tempfile-backed option on one of the experimental branches. I can prioritize that feature if it would be useful to you.

from dedup.

I'd appteciate that a lot, thank you. Also, reallocations? Yikes. Might wanna look at other data structures. I'll shop around.

from dedup.

Here's an idea: when dealing with a mmap(2)ed file, use one of these (especially this) to make a first pass to guess the number of unique lines. Then pre-allocate a hash set with space for that many elements and do the actual deduplication. If there's a more efficient hash set out there, that would be great too.

from dedup.