This is an open source, multi-threaded website crawler written in Python. There is still a lot of work to do, so feel free to help out with development.
Note: This is part of an open source search engine. The purpose of this tool is to gather links only. The analytics, data harvesting, and search algorithms are being created as separate programs.
Web-Scraping A collection of small programs that extract data from a website and packages it to be useful with the use of BeautifulSoup, a Python package for parsing HTML and XML documents. Once you retrive the raw HTML of a site, you can start to select and extract with BeautifulSoup, which parses raw HTML strings and produces an object that mirrors HTML documents' structure. The Rules of Scraping Check a website's Term and Conditions before scraping it and read the statements about legal use of the data. Do not request data from the website too aggressiely and ensure that your program behaves in a reasonable manner. Revisit the website and rewrite code as needed as the layout of the site may change.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent