natzar / pythonsimplecrawlerbot Goto Github PK
View Code? Open in Web Editor NEWA Simple Crawler Bot written in Python. This script is a version of the one used at Domstry.com. You can use it to crawl the internet. It fetches domains, store the data, and adds the links found in current domain to the queue.
Home Page: https://www.domstry.com
License: GNU General Public License v3.0