A fast go based duplicate domains remover

TL;DR

Give it list of URL's and it will find the sites that have similar source code and remove them

Download and Build it using following command:

$ git clone https://github.com/0xsapra/dedupe
$ cd dedupe
$ go build dedupe.go  

$ echo "https://site.com\nhttps://site2.com\nhttps://site.com/asdf.php\n" | ./dedupe  

OR

$ echo "https://site.com\nhttps://site2.com\nhttps://site.com/asdf.php\n" > domains.txt

$ cat domains.txt | ./dedupe

OR, use it will other tools. Like projectdiscovery's httpx. https://github.com/projectdiscovery/httpx and, tomnonnom's waybackurls https://github.com/tomnomnom/waybackurls

$ cat domains.txt | waybackurls | httpx | ./dedupe