Make a stocktake of files and separate work to figure out duplicates with different methods.
This is a bit of a failed project here but I succeeded to a reasonable extent offline, using AHK, Python, SQLLite, and self-created JSON files and SQL insert scripts. The solution was different to what is here but messy and personalised. This project is abandoned from sharing for now.