mrighele / zfs-find-deduped Goto Github PK

These should probably be changed to use a streaming object, not split.

AFAIK, split will
a. block until it has its entire input (which is really huge on any real filesystem), and
b. then create the split array in memory (thereby doubling the already excessive RAM requirement).

Here, you just want to process it as a stream as fast as it comes in, since you´re not doing anything that requires the whole thing at once.

I don't know python well enough to suggest how to do it.

In Ruby, it'd be IO#gets (which is also in anything descending from IO, like File).

In plain Bash it'd be

while IFS='$\n' read -r line
do
  # do something with $line
done < process

This should probably be reversed — there should be fewer dedup blocks than files, at least if you're only keeping ones where the block is used more than once — and then feed the resulting blocks-of-interest list to find_file_indirect_blocks to use as a stream filter, i.e. so that files not on the list are ignored (& not kept in memory).

Fixing these two things should vastly improve the efficiency of the script.