Python implementation of web crawler that searches a given website for an word
Web crawler is a program that browses the World Wide Web in a methodical fashion collecting information when based on the requirement. A crawler collects web page content and links to other web pages from the website.
The spider function looks into the website that the user inputs. Every time, it collects two data, the text on the page and links on the page. If the word is not found in the text of the page, the bot goes to the link of the page and repeats searching for the text. This is done based on the limit provided by the programmer. We have provided 100 pages as the limit.
Source : http://www.netinstructions.com/how-to-make-a-web-crawler-in-under-50-lines-of-python-code/