Please consider an option to treat files as different unless they have the same permissions, owner/group, and modification time.

Simplified use case as an example: a UNIX/Linux system with a stock ".bash_profile" in each user's home directory. These are not really duplicates because they each have their own owner/group, permissions, and date/time metadata.

At the moment I can use rdfind to generate a list of files that are duplicates by data, but I then have to post-process the results.txt file to determine whether or not they are true duplicates.

I have a huge collection of files that is already largely indexed, it would be helpful if rdfind could ask a subprocess about a files rather than actually needing to calculate it at each run.

Hi,

not sure if I am doing it right but If I tell rdfind to scan folders with spaces it fails.

admin@VMX-Mate-finddupes:~$ rdfind -n true /mnt/nas/Adults/TV\ Shows
(DRYRUN MODE) Now scanning "/mnt/nas/Adults/TV Shows", found 0 files.
(DRYRUN MODE) Now have 0 files in total.
(DRYRUN MODE) Removed 0 files due to nonunique device and inode.

if I start rdfind in the dir with rdfind ./ it works but then it cant write the results.txt

Is there an option with -outputname to have the results.txt written to a different location eg ~/results.txt?

thanks
chris

When compiling under Cygwin with the latest code from git (as of August 10, 2020), I get:

Fileinfo.cc:279:14: error: ‘symlink’ was not declared in this scope

Full session:

$ ./bootstrap.sh
it seems like everything went fine. now try
./configure && make

$ ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for g++... g++
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.exe
checking for suffix of executables... .exe
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking whether make supports the include directive... yes (GNU style)
checking dependency style of g++... gcc3
checking whether make sets $(MAKE)... (cached) yes
checking how to run the C++ preprocessor... g++ -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
no
checking whether to enable assertions... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking nettle/sha.h usability... yes
checking nettle/sha.h presence... yes
checking for nettle/sha.h... yes
checking for main in -lnettle... yes
checking for stat... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
configure: checking for c++ 11, disable with --disable-cppstandardcheck
checking whether g++ supports C++11 features with -std=c++11... yes
checking check for fallthrough support... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating config.h
config.status: executing depfiles commands

$ make
make  all-am
make[1]: Entering directory '/home/▒▒▒/build/rdfind'
g++ -std=c++11 -DHAVE_CONFIG_H -I.     -g -O2 -MT rdfind.o -MD -MP -MF .deps/rdfind.Tpo -c -o rdfind.o rdfind.cc
mv -f .deps/rdfind.Tpo .deps/rdfind.Po
g++ -std=c++11 -DHAVE_CONFIG_H -I.     -g -O2 -MT Checksum.o -MD -MP -MF .deps/Checksum.Tpo -c -o Checksum.o Checksum.cc
mv -f .deps/Checksum.Tpo .deps/Checksum.Po
g++ -std=c++11 -DHAVE_CONFIG_H -I.     -g -O2 -MT Dirlist.o -MD -MP -MF .deps/Dirlist.Tpo -c -o Dirlist.o Dirlist.cc
mv -f .deps/Dirlist.Tpo .deps/Dirlist.Po
g++ -std=c++11 -DHAVE_CONFIG_H -I.     -g -O2 -MT Fileinfo.o -MD -MP -MF .deps/Fileinfo.Tpo -c -o Fileinfo.o Fileinfo.cc
Fileinfo.cc: In lambda function:
Fileinfo.cc:279:14: error: ‘symlink’ was not declared in this scope
  279 |       return symlink(target.c_str(), filename.c_str());
      |              ^~~~~~~
Fileinfo.cc: In instantiation of ‘int {anonymous}::transactional_operation(const string&, const Func&) [with Func = Fileinfo::makesymlink(const Fileinfo&)::<lambda(const string&)>; std::string = std::basic_string<char>]’:
Fileinfo.cc:280:6:   required from here
Fileinfo.cc:249:20: error: void value not ignored as it ought to be
  249 |   const int ret = f(filename);
      |                   ~^~~~~~~~~~
make[1]: *** [Makefile:657: Fileinfo.o] Error 1
make[1]: Leaving directory '/home/▒▒▒/build/rdfind'
make: *** [Makefile:536: all] Error 2

It would make sense to add BLAKE3 support since it's way faster than the currently supported hash functions, see https://github.com/BLAKE3-team/BLAKE3 for more info

Hello,

I need to deduplicate a huge mail folder from a mail server and some files are identical in a lot of mailboxes but they should not be hardlinked. I looked at the source code and I look quite easy to filter the list of files with a regex.

Would you consider merging those changes ? If you do I can implement that carefully and make a merge request but if you don't I just make the change I need quick & dirty

I just saw the issue about filtering the answer by file name. So I guess the answer is probably no

Sometimes it is of interest to ignore small files. rdfind already ignores empty files by default. The suggestion is to replace the -ignoreempty flag (or complement it) with a -minsize N flag where N is the minimum size in bytes. Using N=0 would mean empty files are included.

This is an old issue, reported by Andrew Buehler 20131130.

BLAKE2 (512 bit) hashes about 20% faster than fastest sha, maybe worths a try.

I tried rdfind with firmware update files. The problem with these is that the first 1000 Bytes are identical and even the last 64 (current default in Fileinfo.hh) don't differ that much between the files. So rdfind resorts to calculating checksums which takes a long time (large files) in comparison.

Could you maybe add a parameter to set the SomeByteSize value to higher values?

There are spaces missing after the full stop before the last numbers.

Now eliminating candidates based on first bytes:removed 24347 files from list.33115 files left.
Now eliminating candidates based on last bytes:removed 29203 files from list.3912 files left.
Now eliminating candidates based on sha1 checksum:removed 3896 files from list.16 files left.

I think the space needs to be inserted somewhere here.

Enable rdfind to be re-initiated on large jobs.

Right now, trying to scan an 8TB cold storage drive with Linux OS backups. Taking literal days and I've had to kill the job because there's no way to tell how it's making progress.

Hi,

I have the need to frequently check a bunch of virtual images for duplicates.

See below an example for a small data set.

Now have 324 files in total.
Removed 179 files due to nonunique device and inode.
Now removing files with zero size from list...removed 3 files
Total size is 3205167303976 bytes or 3 TiB
Now sorting on size:removed 30 files due to unique sizes from list.112 files left.
Now eliminating candidates based on first bytes:removed 6 files from list.106 files left.
Now eliminating candidates based on last bytes:removed 2 files from list.104 files left.
Now eliminating candidates based on sha1 checksum:

The problem is these are quite huge but often the first and last bytes are equal, so rdfind goes into full checksum calculation.

Secondary problem is that most of these files are sparse files, which means reading them full reads a lot of zeros.

Would it be possible in addition/instead of the first/last bytes to check the first/last x megabytes? Say 1 meg at the beginning would be sufficient, as this either covers a file system or swap space, which is more likely not equal in case the images differ.

It would mean to sometimes read the x MB in addition, but for my use case it would dramatically reduce the total amount of data read.

Hi,
there's is a "problem" with rdfind, creating symlinks and midnight commander. If rdfind is creating a symlink for a duplicate file in the same path, midnight commander IS NOT showing both of them, but only either the symlink OR the real file. OK, by
find <path> -type l -print
The symlinks can be sorted out but this is problematic. How to circumvent that? Check wether symlink should only be created in a new directory (e.g. 000-symlinks in that folder?)

I just ran your program on a 10 Tb backup I made to try to reduce the overall size. Surprised to find that the "failed to make symlink error" hasn't been fixed in the version that apt installs. Seems like something that should be passed to all the automatic installation services....

Hej, Paul! Excellent tool! (I was actually about to start writing something very similar, but now I've found I don't have to! Great!)

My situation is this: I am using rdfind to recover space on my hard drive. So, I run rdfind . in my home directory to see what's going on. The result is that I have a lot of mp3 files in my Downloads directory and other places on my hard drive that iTunes has copied into itself. Bad iTunes! However, I also have a lot of .py and .js files that are used by virtualenv and npm for my various projects.

I would like to remove the duplicated mp3 files but leave the py and js files untouched. I've read the documentation, but I don't see a way to do this currently in rdfind.

I'm imagining the interface might be something like:

rdfind --include-extention mp3 or rdfind --exclude-extention py,js.

This is suggested by Debian bug 815120
and a good idea.

Could you show a simple progress indicator for each of the steps?

For example this takes quite long:

Now eliminating candidates based on sha1 checksum:

You could display reading file i of n until it is finished. Also an elapsed time would be helpful.

This is a documentation change.
See Debian bug 901423

Having a test that tests that the ranking between files (as described in the man page) works as promised would make it easier to do refactoring and cleanups.

Hi there,

there is an error when using configure and i can't figure out what the problem ist.
I used the latest release as well as the latest github master branch.
On the gcc side i tried version 7.4 and 8.3.
What am i missing?

checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... no
checking whether make supports nested variables... yes
checking for g++... g++
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking for style of include used by make... none
checking dependency style of g++... none
checking whether make sets $(MAKE)... (cached) no
checking how to run the C++ preprocessor... g++ -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
no
checking whether to enable assertions... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking nettle/sha.h usability... yes
checking nettle/sha.h presence... yes
checking for nettle/sha.h... yes
checking for main in -lnettle... yes
checking for stat... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
./configure: line 4230: syntax error near unexpected token `11,noext,mandatory'
./configure: line 4230: `      AX_CXX_COMPILE_STDCXX(11,noext,mandatory)'

I've noted that there seems to be btrfs deduplication in the roadmap. I would suggest that it might be easier to provide an option for a sub-process which performs the deduplication on rdfind's behalf.

Then regardless of system configuration a programmer may plug in a tool that simply does whatever "deduplication" is on their filesystem.

There might be no way to do it without breaking backwards compatibility, but specifying options to rdfind is really strange as currently implemented.

Long options should really be specified as
rdfind --makehardlinks <path>
instead of
rdfind -makehardlinks true <path>

To be consistent with other command line tool invocations.

It seems like the -minsize argument is ignored in the commnad:

rdfind -removeidentinode false -minsize 100000000 -ignoreempty true -makeresultsfile true -makehardlinks true /tank/

Tons of 16 byte files and the like have been hardlinked.

Compiler warnings are very useful and fixing them makes the code better.

For reference, see this: c++ best practices

When a directory is mounted over smb via samba on linux, running rdfind inside that directory spams:

recursion limit exceeded
recursion limit exceeded
recursion limit exceeded
recursion limit exceeded
recursion limit exceeded

and never completes.

Observed the following errors in the kernel log:

[ 1588.565758] CIFS VFS: Close unmatched open
[ 1588.566208] CIFS VFS: Close unmatched open
[ 1588.566521] CIFS VFS: Close unmatched open
[ 1588.566856] CIFS VFS: Close unmatched open
[ 1588.567192] CIFS VFS: Close unmatched open
[ 1588.567573] CIFS VFS: Close unmatched open
[ 1588.567809] CIFS VFS: Close unmatched open
[ 1588.568044] CIFS VFS: Close unmatched open
[ 1588.568425] CIFS VFS: Close unmatched open
[ 1588.568691] CIFS VFS: Close unmatched open

When a directory is mounted in wsl1 over smb via samba, it also includes errors such as:

Dirlist.cc::handlepossiblefile: This should never happen. FIXME! details on the next row:
possiblefile="./master/.hardinfo"
Dirlist.cc::handlepossiblefile: This should never happen. FIXME! details on the next row:
possiblefile="./master/build/build/ouya/menu/initrd/data"

along with the recursion limit exceeded errors.

Samba host is ubuntu 20.04, running kernel 5.6.11.
Linux client is ubuntu 19.10.
WSL client is ubuntu 18.04.
"This is rdfind version 1.4.1"
Issues do not manifest when running over nfs.

Any Mac users? I would be happy if someone could help me adding mac builds to travis.

There is probably quite a lot of "gnu-isms" in the shell scripts etc which may be a problem.

I use rdfind to make hardlinks between two big directory trees. The number of files is ~25000. Files are immutable - they can be created and deleted but not modified.

Each time I run rdfind it excludes about 10% of files on some criteria but after that it starts the most time consuming part - read first bytes, read last bytes. This takes a lot of time because the number of files to read is more than 20000 and the total size of files is about 40GB.

My idea is to cache file checksums somewhere so when I start rdfind next time it can read checksums of old files from the cache and calculate checksums only for those files which are new (not existed at the moment of previous run).

I believe this can dramatically reduce execution time.

What I have done to resolve this but failed:

• rdfind /Volumes/1TB\ HDD2/Pictures/
• Move copies of jpg files to root Directory of 1TB HDD2
• Change from jpg to jpeg.

Output:
❯ rdfind /Volumes/1TB\ HDD2/Pictures/ Now scanning "/Volumes/1TB", found 0 files. Now scanning "HDD2/Pictures", found 0 files. Now have 0 files in total. Removed 0 files due to nonunique device and inode. Total size is 0 bytes or 0 B Removed 0 files due to unique sizes from list.0 files left. Now eliminating candidates based on first bytes:removed 0 files from list.0 files left. Now eliminating candidates based on last bytes:removed 0 files from list.0 files left. Now eliminating candidates based on sha1 checksum:removed 0 files from list.0 files left. It seems like you have 0 files that are not unique Totally, 0 B can be reduced. Now making results file results.txt

MacOS Catalina
Seagate 1TB

The error message when an action has failed is subpar. See Debian bug 648671
This is related to, and perhaps partially solved by, #5.

I might be misled, but I expected the run of rdfind . -makehardlinks to be idempotent, on a file system that hasn't changed meanwhile. So a second run should not be able to replace any files with hard links.
But the output suggests, that it isn't. See my output of consecutive runs below, running rdfind version 1.3.5 on Ubuntu 18.04 (latest stable).

I am now afraid, that unique files are affected and I'm unsure if I should restore backup.
Please, tell me, where I have a misconception.

stefan@meidling:/var/phoenix/pan-uploads/images/episode/4/7$ sudo rdfind . -makehardlinks true
Now scanning ".", found 50145 files.
Now have 50145 files in total.
Removed 1830 files due to nonunique device and inode.
Now removing files with zero size from list...removed 0 files
Total size is 812738601 bytes or 775 MiB
Now sorting on size:removed 5608 files due to unique sizes from list.42707 files left.
Now eliminating candidates based on first bytes:removed 4329 files from list.38378 files left.
Now eliminating candidates based on last bytes:removed 4885 files from list.33493 files left.
Now eliminating candidates based on md5 checksum:removed 527 files from list.32966 files left.
It seems like you have 32966 files that are not unique
Totally, 442 MiB can be reduced.
Now making results file results.txt
Now making hard links.
Making 28744 links.

stefan@meidling:/var/phoenix/pan-uploads/images/episode/4/7$ sudo rdfind . -makehardlinks true
Now scanning ".", found 50146 files.
Now have 50146 files in total.
Removed 30259 files due to nonunique device and inode.
Now removing files with zero size from list...removed 0 files
Total size is 356656677 bytes or 340 MiB
Now sorting on size:removed 7452 files due to unique sizes from list.12435 files left.
Now eliminating candidates based on first bytes:removed 5760 files from list.6675 files left.
Now eliminating candidates based on last bytes:removed 5513 files from list.1162 files left.
Now eliminating candidates based on md5 checksum:removed 532 files from list.630 files left.
It seems like you have 630 files that are not unique
Totally, 4 MiB can be reduced.
Now making results file results.txt
Now making hard links.
Making 315 links.

stefan@meidling:/var/phoenix/pan-uploads/images/episode/4/7$ sudo rdfind . -makehardlinks true
Now scanning ".", found 50145 files.
Now have 50145 files in total.
Removed 30378 files due to nonunique device and inode.
Now removing files with zero size from list...removed 0 files
Total size is 351620489 bytes or 335 MiB
Now sorting on size:removed 7518 files due to unique sizes from list.12249 files left.
Now eliminating candidates based on first bytes:removed 5810 files from list.6439 files left.
Now eliminating candidates based on last bytes:removed 5515 files from list.924 files left.
Now eliminating candidates based on md5 checksum:removed 532 files from list.392 files left.
It seems like you have 392 files that are not unique
Totally, 2 MiB can be reduced.
Now making results file results.txt
Now making hard links.
Making 196 links.

stefan@meidling:/var/phoenix/pan-uploads/images/episode/4/7$ sudo rdfind . -makehardlinks true
Now scanning ".", found 50146 files.
Now have 50146 files in total.
Removed 30434 files due to nonunique device and inode.
Now removing files with zero size from list...removed 0 files
Total size is 350678785 bytes or 334 MiB
Now sorting on size:removed 7548 files due to unique sizes from list.12164 files left.
Now eliminating candidates based on first bytes:removed 5831 files from list.6333 files left.
Now eliminating candidates based on last bytes:removed 5521 files from list.812 files left.
Now eliminating candidates based on md5 checksum:removed 532 files from list.280 files left.
It seems like you have 280 files that are not unique
Totally, 1 MiB can be reduced.
Now making results file results.txt
Now making hard links.
Making 140 links.

stefan@meidling:/var/phoenix/pan-uploads/images/episode/4/7$ sudo rdfind . -makehardlinks true
Now scanning ".", found 50146 files.
Now have 50146 files in total.
Removed 30460 files due to nonunique device and inode.
Now removing files with zero size from list...removed 0 files
Total size is 350342492 bytes or 334 MiB
Now sorting on size:removed 7566 files due to unique sizes from list.12120 files left.
Now eliminating candidates based on first bytes:removed 5839 files from list.6281 files left.
Now eliminating candidates based on last bytes:removed 5521 files from list.760 files left.
Now eliminating candidates based on md5 checksum:removed 532 files from list.228 files left.
It seems like you have 228 files that are not unique
Totally, 1 MiB can be reduced.
Now making results file results.txt
Now making hard links.
Making 114 links.

stefan@meidling:/var/phoenix/pan-uploads/images/episode/4/7$ sudo rdfind . -makehardlinks true
Now scanning ".", found 50146 files.
Now have 50146 files in total.
Removed 30477 files due to nonunique device and inode.
Now removing files with zero size from list...removed 0 files
Total size is 350052283 bytes or 334 MiB
Now sorting on size:removed 7574 files due to unique sizes from list.12095 files left.
Now eliminating candidates based on first bytes:removed 5846 files from list.6249 files left.
Now eliminating candidates based on last bytes:removed 5523 files from list.726 files left.
Now eliminating candidates based on md5 checksum:removed 532 files from list.194 files left.
It seems like you have 194 files that are not unique
Totally, 920 KiB can be reduced.
Now making results file results.txt
Now making hard links.
Making 97 links.

stefan@meidling:/var/phoenix/pan-uploads/images/episode/4/7$ sudo rdfind . -makehardlinks true
Now scanning ".", found 50146 files.
Now have 50146 files in total.
Removed 30490 files due to nonunique device and inode.
Now removing files with zero size from list...removed 0 files
Total size is 349873207 bytes or 334 MiB
Now sorting on size:removed 7581 files due to unique sizes from list.12075 files left.
Now eliminating candidates based on first bytes:removed 5850 files from list.6225 files left.
Now eliminating candidates based on last bytes:removed 5525 files from list.700 files left.
Now eliminating candidates based on md5 checksum:removed 532 files from list.168 files left.
It seems like you have 168 files that are not unique
Totally, 749 KiB can be reduced.
Now making results file results.txt
Now making hard links.
Making 84 links.

Provide the ability to tell rdfind not to parse part of a tree...

e.g. rdfind -makehardlinks true "/usr/me" -omit "/usr/me/somedirectory"

In my use-case I have a directory with 30 sub-directories... and I need to run rdfind across 29 of the 30.

I use ZFS to store all my files in a big pool. Sometimes I have duplicates I identify with rdfind in a dry-run, check the results.txt manually and (if it's ok with me) re-run rdfind to really delete the duplicate files. So rdfind not only needs to fully read the remaining files in full twice (at least) to compute the checksums, but also it does not leverage the block checksums of every file that are an inherent feature of ZFS (and calculated anyway, but at write-time of the file).

The ZFS command zdb gives an indication on how this could work. To query which files (and their ZFS object ID) are on a given ZFS file system (here minitank/fw/video/gopro):

$ zdb -vvv minitank/fw/video/gopro | grep \(type:\ Regular | head -n 100
(...)
		.VolumeIcon.icns = 7 (type: Regular File)
		com.apple.FinderInfo = 9 (type: Regular File)
		VolumeConfiguration.plist = 16 (type: Regular File)
(...)
		GOPR0410.MP4 = 110 (type: Regular File)
		GOPR0428.LRV = 109 (type: Regular File)
		G0030427.JPG = 108 (type: Regular File)
		GOPR0434.MP4 = 121 (type: Regular File)
		G0010416.JPG = 120 (type: Regular File)
		GOPR0433.MP4 = 116 (type: Regular File)
		G0020421.JPG = 115 (type: Regular File)

This allows to query (for example) what the checksums for each block the ZFS object 115 (file G0020421.JPG) has:

$ zzdb -vvvvv minitank/fw/video/gopro 115 | grep "L0 ZFS plain file" | grep -o cksum=.*
cksum=2b0ad5f93fb4:7298cf4a6946d86:9121b965256654b8:1bc1f6f4a2a099d8
cksum=4097453f3d6c:103cb4291526d015:8afcb0a258e95b77:a9e71a3e8f82f19d
cksum=4022edcd7609:100fea58fa9ac828:2ca5db609ae0a854:faea7f497f7daf2c
cksum=404ea3b38c21:1015c2c361081102:bcc31ed291dca604:4b126a2d4052739b
cksum=3f598a2ab67a:fde544c32e664c2:abd956aeed12b9db:7cf2e4d2a6c9bd4f
cksum=3f675000fe35:fd98072b0095081:5a98cae21efdd757:f7fbf8cd6fa73304
cksum=3ef7cee7b23e:fcc06f4ad0f2b00:9588581ab10b1fe6:c6ee212e369f0f79
cksum=3e4bcddec672:f94b6b8790382e5:628b25de42d08d2a:b563887a5e7995a8
cksum=3d43f9295614:f4fc8a2c9ced025:9bf45e95e78116e6:97f66dbb9bcc3da7
cksum=3db1b8dcb8ea:f6898164ef8961d:717da2e5f2a8ae7c:7d1ff9f6e079a349
cksum=3e8278d5f3eb:f969929d7718668:cc9012a3cdbee3ab:be677c0b5973d2e
cksum=3edd74e3736a:fb28bbd504ea3ed:29d1881b6fdf791f:d88937b1fa135fe6
cksum=3e6d420fa43c:f925adf1e0beeae:e6ec2c707a56118d:2b704c513ab8e7a4
cksum=3df8ee864c70:f7c1a08ad5774eb:9f92915c010af70:37f591cd192339ae
cksum=3d95d67f89ef:f60c81b1766b80c:a5ffc708f633a88f:a4fa96a5806af28d
cksum=3e191cd0dcf0:f7a200c5a8e12ce:dae62952d2d776eb:a932d6341e2bf523
cksum=409332d2588d:ffff751b79ce0c4:13eb43c9f54317cc:352b881aa6025856
cksum=53e67ac1208:283c714f9b3a2a1:78069759bb33e8b1:1e280f42ef6e382a

Now for rdfind using the zdb command is not a very good idea (except for a PoC maybe), as the output format is clearly not meant for automatic processing (it is also said not to be backward compatible through the ZFS releases). But zdb has no real magic, AFAIK it just queries the ZFS API and spits out the resulting information as text.

So when using ZDB, reading the actual files to check for equality would be unnecessary - which would render rdfind even quicker.

In a small experiment I could reproduce the cksum values being stable across different ZFS file systems (bar the checksums of the last block, because of a shorter block length on one of the ZFS file systems) and also intra-file system with a simple copy of the file (but different name and access times etc.).

See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=795790

See also issue #10 for tests related to this.

Rdfind 1.4.0 failed to build on some of the Debian's build machines. It worked on amd64, arm64, mips64, ppc64el, s390x, alpha, ia64, ppc64, riscv64 and sparc64. It failed on armel, armhf, i386,mips,hppa, hurd-i386,m68k, powerpc, powerpcspe, sh4 and x32. The common denominator is that all those are (I believe) 32 bit platforms.
This demonstrates that building on a single architecture (amd64) is not enough.
Here are some ideas how to fix this on appveyour and/or travis

• build on 32 bit x86 (how? through debian multiarch?)
• qemu ?
• docker?
• x32?

Is there any freely available continuous integration that uses arm? Thankful for input and suggestions.

The dryrun option is possible to misunderstand, and the error message is bad. See Debian bug 754663 and Ubuntu bug 1782273.

It is perhaps difficult to get right from glancing over the documentation because it needs a boolean argument true or false afterwards. This is a bad design (sorry!) but must remain as is for backwards compatibility. Making the true|false optional, would lead to ambiguity if invoking like this awkward example shows:

cd $(dirname $(which true))
rdfind -dryrun true false

It would be good to improve the error output, such that the error message immediately suggest what should be done.
Perhaps the builtin documentation can be made a bit clearer.

I'm trying to build this on centos 7 with autotools. I have an old version of rdfind with yum.

checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking nettle/sha.h usability... no
checking nettle/sha.h presence... no
checking for nettle/sha.h... no
configure: error:
 nettle header files missing. Please install nettle
 first. If you have already done so and get this error message
 anyway, it may be installed somewhere else, maybe because you
 don't have root access. Pass CPPFLAGS=-I/your/path/to/nettle to configure
 and try again. The path should be so that #include "nettle/sha.h" works.
 On Debian-ish systems, use "apt-get install nettle-dev" to get a system
 wide nettle install.

I tried the pass CPPFLAGS=-I"/opt/rfind/nettle" ./configure ... no cigar. sha.h is there and nettle seemed to build correctly.

When running autoreconf -fi and then ./configure I get the following error

./configure: line 4176: syntax error near unexpected token `11,noext,mandatory'
./configure: line 4176: `      AX_CXX_COMPILE_STDCXX(11,noext,mandatory)'

Probably the macro is missing? Creating a local m4 folder might help?

After the run, I see no additional free space, despite the Totally, 740 Gib can be reduced in the console.
Here is my full output:

# df -h /mnt/s4; time rdfind -removeidentinode false -makehardlinks true /mnt/s4/; df -h /mnt/s4
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdf1       7.3T  7.3T     0 100% /mnt/s4

Now scanning "/mnt/s4", found 436762 files.
Now have 436762 files in total.
Now removing files with zero size from list...removed 1008 files
Total size is 9425999506070 bytes or 9 Tib
Now sorting on size:removed 142093 files due to unique sizes from list.293661 files left.
Now eliminating candidates based on first bytes:removed 117002 files from list.176659 files left.
Now eliminating candidates based on last bytes:removed 6037 files from list.170622 files left.
Now eliminating candidates based on md5 checksum:removed 13035 files from list.157587 files left.
It seems like you have 157587 files that are not unique
Totally, 740 Gib can be reduced.
Now making results file results.txt
Now making hard links.
Making 111390 links.

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdf1       7.3T  7.3T  620K 100% /mnt/s4

Seems I'm using the wrong flags, any advice?

Find and fix any issues preventing rdfind from being reproducibly built. See

• https://reproducible-builds.org/
• https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/rdfind.html

It's not clear what happens if you run this on an NTFS volume mounted from Linux. Will NTFS hard links be created?

Running 1.4.1 from the Ubuntu package. Works fine except when encountering unreadable subdirectories:

tmp$ mkdir rdfind-test
tmp$ sudo mkdir rdfind-test/subdir
tmp$ sudo touch rdfind-test/subdir/file
tmp$ mkdir rdfind-test/otherdir
tmp$ touch rdfind-test/otherdir/file
tmp$ rdfind rdfind-test/
Now scanning "rdfind-test", found 0 files.
Now have 0 files in total.
Removed 0 files due to nonunique device and inode.
Total size is 0 bytes or 0 B
Removed 0 files due to unique sizes from list.0 files left.
Now eliminating candidates based on first bytes:removed 0 files from list.0 files left.
Now eliminating candidates based on last bytes:removed 0 files from list.0 files left.
Now eliminating candidates based on sha1 checksum:removed 0 files from list.0 files left.
It seems like you have 0 files that are not unique
Totally, 0 B can be reduced.
Now making results file results.txt
tmp$ sudo chmod go-rwx rdfind-test/subdir/
tmp$ rdfind rdfind-test/
Now scanning "rdfind-test"Dirlist.cc::handlepossiblefile: This should never happen. FIXME! details on the next row:
possiblefile="rdfind-test/subdir"
, found 0 files.
Now have 0 files in total.
Removed 0 files due to nonunique device and inode.
Total size is 0 bytes or 0 B
Removed 0 files due to unique sizes from list.0 files left.
Now eliminating candidates based on first bytes:removed 0 files from list.0 files left.
Now eliminating candidates based on last bytes:removed 0 files from list.0 files left.
Now eliminating candidates based on sha1 checksum:removed 0 files from list.0 files left.
It seems like you have 0 files that are not unique
Totally, 0 B can be reduced.
Now making results file results.txt

How to hardlink duplicates on whole filesystem?

This is after suggestion from SB sending me an email, thanks!

While waiting for rdfind to complete, it would be nice to present some kind of feedback in case the session is interactive. During the first step (scanning the inputs), it is difficult to know how long time is left, but the later steps (reading first bytes, and hashing the contents) it is certainly possible.

The downside is that it complicates the program and slows it down, so it should probably be made an opt-in.

It may also be confusing for users that the progress bar is per step. Perhaps it could say
Step 1/3: ###
and then once that step is finished, the output will look like:
Step 2/3: #

This is the proposal:

Alternative 1 - progress bar

The progress bar could be a character like # (like wget does, but without the percentage and time estimate). The output is # repeated 0 to 80 times, depending on the progress.

1. During scanning which files exist, present nothing.
2. During scanning the first bytes of each file, output progress as i/N where i is the current step and N the number of files known to be scanned
3. During hashing the file contents, output progress as i/N where i is the current step and N the number of files known to be hashed

This will be misleading in case some of the files are unevenly fast to read (one dir on a local ssd, the other on a network share), or the sizes vary a lot.

Alternative 2 - spinner

A spinner alternates between displaying the characters \ | / -, making it appear like a spinning rod.

1. During scanning which files exist, turn the spinner each time a file is found.
2. During scanning the first bytes of each file, output progress as i/N where i is the current step and N the number of files known to be scanned
3. During hashing the file contents, output progress as i/N where i is the current step and N the number of files known to be hashed

The good thing with this is that it is easy to see the process has not hang, and perhaps a bit less visually obstructive than the previous alternative.

Implementation

This does not look difficult to implement, see for instance https://stackoverflow.com/a/14539953

Feedback on this proposal

Feel free to drop your comments and suggestions here!

As described in readme, currently identical inodes can be dealt with in one of two ways:

• with -removeidentinode true, one might need to run rdfind several times to merge two hardlinked groups of identical files (each run will do md5 computation on these two inodes)
• with -removeidentinode false, this can be done in one pass, at expense of doing md5 computation on each of hardlinks (I assume so).

I suggest improving this in such a way that instead of "collapsing a group of hardlinked files to a single entry" and dealing with this entry alone, on step 5 of the algorithm we keep a list of all hardlinks linked to this entry and, after step 12 on the algorithm (after checksum comparison), if "main entry" is still in list of duplicates - add all its hardlinks to the list, too

While using rdfind to clean up my ebooks, the tool deleted copies that were downloaded first in favor of those that had a (1).ext in the filename, an artifact of downloading files twice in Firefox.

In the -deleteduplicates process, dropping files that have already been tagged by another program as potential dupes should be preferable.

DUPTYPE_FIRST_OCCURRENCE 69 0 6487394 64769 29229369 3 eBooks/9781787123687(1).epub
DUPTYPE_WITHIN_SAME_TREE -69 0 6487394 64769 29230708 3 eBooks/9781787123687(2).epub
DUPTYPE_WITHIN_SAME_TREE -69 0 6487394 64769 29231261 3 eBooks/9781787123687.epub
DUPTYPE_FIRST_OCCURRENCE 310 0 7356520 64769 28967090 3 eBooks/gamehacking(1).pdf
DUPTYPE_WITHIN_SAME_TREE -310 0 7356520 64769 25559164 3 eBooks/gamehacking.pdf
DUPTYPE_FIRST_OCCURRENCE 28 0 7367823 64769 33296269 3 eBooks/9781783553358-PYTHON_DATA_ANALYSIS(1).pdf
DUPTYPE_WITHIN_SAME_TREE -28 0 7367823 64769 33296270 3 eBooks/9781783553358-PYTHON_DATA_ANALYSIS.pdf

Many filesystems have 16bit (or even smaller) link counters, eg. ext4 has a limit of 65000 links per file. If there are more duplicate files than that, rdfind tries to link all of them into one, and it will fail to make links after the limit has been reached.

If the limit has been reached, the next processed copy of the file should be used as a target for subsequent duplicates instead of replacing it with a link (and so on if the limit is reached again).

The maximum number of links for any given file or dir should be able to be found with pathconf/fpathconf and the current link count with stat/fstat

Support for min and max filesizes to allow for “file size based partitioning” of the deduplication process. One could think of this as “banding” along the spectrum of sizes or sharding.

Why this enhancement is safe:

The size filtering only affects whether or not files are pushed onto the queue to be worked on. It is merely a “cull” of the files that are walked in the directory tree..

Why this enhancement is needed:

We work on several machine learning and deep learning projects which operate on millions of files. Deduplication is critical in certain applications and categorizations. However running rdfind on the entire dataset can lead to jobs that do not finish before the arrival of new data and the performance degrades as the working set grows beyond physical memory. By partitioning on file size, the deduplication operation can be run in parallel (machines and cores) and in smaller batches with no loss of functionality and gains in speed from parallelization. Additionally, since certain size files are more frequent in our applications, it is possible to target deduplication to “the most active file sizes” in the archives.

Other motivations:

I have used this feature for some years and it is valuable to me. I think they would be useful to others. There is no discernible impact on runtime performance. Arguments that running in parallel is fraught with danger only apply if running in parallel over the same size band which is no different than running two rdfind jobs at once. (This by the way happens when using cron and discovering a prior job has not finished, so partitioning to faster smaller jobs is actually safer in my mind.)

Working Implementation Available (see pull request)

Right now "rdfind … -makesymlinks|-deleteduplicates true … /directory1/ /directory2/" will attempt to prune duplicates in both. Sometimes it would be safer or more convenient if there was an option allowing to:
A: keep /directory1/ from any modifications and remove/replace duplicates from /directory2/ only; or
B: simply don't count duplicates within the same tree at all, only between different trees.
Perhaps (B) is better, as it could be used instead of (A) by running twice - 1st pass without this option pruning the directories that don't need to be preserved, then the 2nd pass with "between directories only" option and the one which should be treated as read-only in the first directory argument, to have it always counted as the "original" instance.
Especially since rdfind already discerns "DUPTYPE_WITHIN_SAME_TREE" vs. "DUPTYPE_OUTSIDE_TREE".

When running the script on multiple devices, the following error appears:

failed to make hardlink /mount1/…/file1 to /mount2/…/file2    
Rdutil.cc: Failed to apply function f on it. 

file1 ends up being deleted. There should be some kind of check to verify that both files live in the same device.
I'm running 1.3.5, I haven't checked the git master.