surfingdoggo / hckrnnscrape Goto Github PK

xml.py

example before (xml) http://builds.wbgames.com/free/free-desi-chat-canada.html 2019-09-10 monthly 0.6

xml.py grabs the html file from between the tags

example converted to json (before.json) obj00000000 = { "html":"free-desi-chat-canada.html", "domains": { "domain0":"builds.wbgames.com", } }

scraper.py

iterates through the before.json in json/before.json obj(i=00000000; i++)[index of html file] searches google for this file

searches for results whose url is a subdomain

reads the HTML for # of h2 html tags if it's a match, it writes the domain to the after.json file

instead do:

grabs sub.domain.com adds /sitemap.xml if exists, writes as much in after.json file with the domain if !exists, writes false with domain if exists, downloads it to ./new.sitemaps/