anantakoirala / proxy_web_crawler Goto Github PK

This script automates the process of searching for a website via keyword and the Bing search engine.... page after page

Pass a complete URL and at least 1 keyword as command line arguments:
python proxy_crawler.py -u https://www.example.com -k keyword

python proxy_crawler.py -u https://www.whatsmyip.org -k "my ip"

If on a Linux system, proxy_crawler can run headless. Give the -x option (requires XVFB):
python proxy_crawler.py -u https://www.whatsmyip.org -k "my ip" -x

• It first scrapes a list of proxies from the web using SSL Proxies
• Then using a new proxy socket for each iteration, the specified keyword(s) is searched for via Bing until the desired website is found
• The website is then visited, and one random link is clicked within the website
• The bot is slowed down on purpose
• If searching with multiple keywords, wrap them in quotes: "example search phrase"

    
pip install selenium
apt-get install xvfb (Linux only)