Add Internet Archive Brozzler distributed crawler in Python

Rust based crawler or indexer https://github.com/spider-rs/spider. Also the fastest and most efficient OSS crawler.

Photon is an incredibly fast crawler which extracts URLs, website accounts, emails, files, endpoints and much more.

https://github.com/s0md3v/Photon

Your repo is so cool, so I want to contribute my crawler repo to your awesome list.

My repo is here : https://github.com/khanhicetea/distributed-webcrawler
It's built on-top Docker, Python, Redis and Gearman Job Server with a distributed architecture :)

Hope it will be helpful :)

There is typos in this line of the README.md file:

https://github.com/weaming/picpicgo

Add https://github.com/LAW-Unimi/BUbiNG - The LAW next generation crawler.

Could you add Add Bose Framework to Python Scrapers. It is a web scraping framework.

Hi, great repo!
So I'd like to add this website https://www.free-proxies.info
It's very helpful for crawling because it's grant an API for get free or premium proxy!

There are also some filters like transparent proxy or anonymous; google proxy and more..
It's always need a proxy for crawling the web!

rt.

Jvppeteer is a Java library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol. Jvppeteer runs headless by default, but can be configured to run full (non-headless) Chrome or Chromium
https://github.com/fanyong920/jvppeteer

@BruceDone checkout #62 please, added reference to https://dev.to/spacenomad/mercator-url-frontier-in-golang-15o2

e.g. for R, HTML parsers:

• XML - XML and HTML scraper and parser. Also works via FTP.
• xml2 - R wrapper for the libxml2 C library.

e.g. for R, ghost browsers:

• RSelenium - R bindings for the Selenium 2.0 WebDriver.
• I think RStudio is working on something similar, but cannot find it any more

Link to markdown:
https://gist.github.com/gustavorps/98a9b03143db24c5acedab343b7e7a55

Preview

Hi,
Can we install this crawler using Kali Linux? Also, can we extract the web form field label names from a URL and the sub-links through the main domain, If not, can you suggest one?

Thank you!

Is there a crawler with an web GUI or rather a frontend such as "Octoparse" for Windows oder "Web Alert" for Android?

Here is my crawler made in Rust.
Supports seed database, adds external links to database, simple parsing and adding to elastic search.

https://github.com/tclaus/web_crawler

A program to download photo and video from any site (e.g. Reddit, Twitter, Instagram).

https://github.com/AAndyProgram/SCrawler