Topic: web-scraping Goto Github

👇 Here are 5419 public repositories matching this topic...

a9t9 / rpa
web-scraping,UI.Vision: Open-Source RPA Software (formerly Kantu) - Modern Robotic Process Automation with Selenium IDE++
User: a9t9
Home Page: https://ui.vision/rpa
automation web-automation selenium-ide visual-recognition ui-tests imacros autohotkey sikulix opencv webassembly browser-extension browser-automation data-driven-tests web-scraping uipath

achuthasubhash / complete-life-cycle-of-a-data-science-project
web-scraping,Complete-Life-Cycle-of-a-Data-Science-Project
User: achuthasubhash
feature-engineering unsupervised-learning data-science deep-learning machine-learning exploratory-data-analysis reinforcement-learning eda pytorch supervised-learning transfer-learning nlp-models python-library analysis data-analysis federated-learning dataset python web-scraping scraper

adbar / trafilatura
web-scraping,Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
User: adbar
Home Page: https://trafilatura.readthedocs.io
article-extractor corpus corpus-builder corpus-tools crawler html-to-markdown html2text news news-aggregator news-crawler nlp readability rss-feed scraping tei text-cleaning text-extraction text-mining text-preprocessing web-scraping

alecxe / scrapy-fake-useragent
web-scraping,Random User-Agent middleware based on fake-useragent
User: alecxe
web-scraping python scrapy

alexmathew / scrapple
web-scraping,A framework for creating semi-automatic web content extractors
User: alexmathew
Home Page: http://alexmathew.github.io/scrapple
python css-selector xpath-expression web-scraper web-scraping scrapers scraping scrapy selector extractor

alirezamika / autoscraper
web-scraping,A Smart, Automatic, Fast and Lightweight Web Scraper for Python
User: alirezamika
scraping scraper scrape webscraping crawler web-scraping ai artificial-intelligence python webautomation automation machine-learning

apify / crawlee
web-scraping,Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Organization: apify
Home Page: https://crawlee.dev
web-scraping web-crawling npm headless-chrome puppeteer automation apify scraping crawling crawler headless scraper web-crawler javascript nodejs playwright typescript

apify / crawlee-python
web-scraping,Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Organization: apify
Home Page: https://crawlee.dev/python/
apify automation beautifulsoup crawler crawling headless headless-chrome pip playwright python scraper scraping web-crawler web-crawling web-scraping

austinoboyle / scrape-linkedin-selenium
web-scraping,`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.
User: austinoboyle
selenium selenium-webdriver linkedin scraping web-scraper web-scraping python scrape scraper

codingforentrepreneurs / 30-days-of-python
web-scraping,Learn Python for the next 30 (or so) Days.
User: codingforentrepreneurs
Home Page: https://cfe.sh/projects/30-days-python-38
api automation csv fastapi flask jupyter pandas python python3 rest-api selenium selenium-webdriver tutorial web-scraping

datahenhq / till
web-scraping,DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
Organization: datahenhq
Home Page: https://till.datahen.com
web-scraping man-in-the-middle proxy-server mitm scraping crawler scraper

dgtlmoon / changedetection.io
web-scraping,The best and simplest free open source web page change detection, website watcher, restock monitor and notification service. Restock Monitor, change detection. Designed for simplicity - Simply monitor which websites had a text change for free. Free Open source web page change detection, Website defacement monitoring, Price change notification
User: dgtlmoon
Home Page: https://changedetection.io
website-monitor website-monitoring change-detection monitoring self-hosted change-alert change-monitoring website-change-monitor url-monitor changedetection

dinubs / coolqlcool
web-scraping,Nextjs server to query websites with GraphQL
User: dinubs
Home Page: https://cool.coolql.cool
graphql javascript nextjs schema web-scraping

evil0ctal / douyin_tiktok_download_api
web-scraping,🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
User: evil0ctal
Home Page: https://douyin.wtf
python pywebio tiktok douyin api scraper fastapi no-watermark online-parsing async

gildas-lormeau / single-file-cli
web-scraping,CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)
User: gildas-lormeau
cli nodejs single-file web-archiving web-scraper web-scraping archiving scraping-websites crawler web-crawler

go-rod / rod
web-scraping,A Chrome DevTools Protocol driver for web automation and scraping.
Organization: go-rod
Home Page: https://go-rod.github.io
cdp chrome-headless chrome-devtools chrome-devtools-protocol headless web-scraping automation scraper devtools devtools-protocol

gosom / google-maps-scraper
web-scraping,scrape data data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
User: gosom
golang google-maps-scraping web-scraper web-scraping distributed-scraper distributed-scraping google-maps

intoli / user-agents
web-scraping,A JavaScript library for generating random user agents with data that's updated daily.
Organization: intoli
browser-automation browsers javascript navigator random randomization user-agent user-agent-spoofer web-scraping

jaebradley / basketball_reference_web_scraper
web-scraping,NBA Stats API via Basketball Reference
User: jaebradley
Home Page: http://jaebradley.github.io/basketball_reference_web_scraper/
basketball-reference python nba web-scraping web-scraper

je-suis-tm / web-scraping
web-scraping,Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist
User: je-suis-tm
Home Page: https://je-suis-tm.github.io/web-scraping
sraping scrapper futures futures-historical-data reuters wall-street-journal bloomberg python-web-scraper news-websites financial-times

juancarlospaco / faster-than-requests
web-scraping,Faster requests on Python 3
User: juancarlospaco
Home Page: https://gist.github.com/juancarlospaco/37da34ed13a609663f55f4466c4dbc3e
python python3 python-library http-requests python-requests web-scraping download-file speed curl cython

justmarkham / dat8
web-scraping,General Assembly's 2015 Data Science course in Washington, DC
User: justmarkham
data-science machine-learning scikit-learn data-analysis pandas jupyter-notebook python course linear-regression logistic-regression

lexiforest / curl_cffi
web-scraping,Python binding for curl-impersonate via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
User: lexiforest
Home Page: https://curl-cffi.readthedocs.io/
curl http-client curl-impersonate http https ja3 ja3-fingerprint tls-fingerprint fingerprinting web-scraping

lit26 / finvizfinance
web-scraping,Finviz analysis python library.
User: lit26
Home Page: https://finvizfinance.readthedocs.io/en/latest/
web-scraping fundamental-analysis stock-news technical-analysis inside-trader financial-analysis pandas stock-charts fundament screener outer-ratings pypi crypto forex earnings-calls

lorien / awesome-web-scraping
web-scraping,List of libraries, tools and APIs for web scraping and data processing.
User: lorien
web-scraping captcha-bypass captcha-recaptcha crawling crawling-framework crawling-python crawling-tool scraping scraping-framework scraping-python

lorien / grab
web-scraping,Web Scraping Framework
User: lorien
Home Page: https://grab.readthedocs.io
web-scraping http-client framework python pycurl asynchronous network urllib3 spider crawler crawling scraping python-library python3

mherrmann / helium
web-scraping,Lighter web automation with Python
User: mherrmann
selenium selenium-python python python3 webdriver chrome firefox web-automation web-scraping helium

oxylabs / amazon-scraper
web-scraping,Free Trial Amazon Scraper API for extracting search, product, offer listing, reviews, question and answers, best sellers and sellers data.
Organization: oxylabs
Home Page: https://oxylabs.io/products/scraper-api/ecommerce/amazon?utm_source=git-internal&utm_medium=github&utm_campaign=repo&utm_content=oxylabs/amazon-scraper
amazon-api amazon-scraper amazon-scraping e-commerce-api web-scraping python amazon-scraper-api amazon-scraping-library amazon-price-tracker scraping-api

oxylabs / how-to-scrape-google-scholar
web-scraping,A guide for extracting titles, authors, and citations from Google Scholar using Python and Oxylabs SERP Scraper API.
Organization: oxylabs
Home Page: https://oxylabs.io/products/scraper-api/serp
google-scholar google-scholar-scraper python python-scraper scraper-api web-scraper web-scraping google-scholar-scrapper google-search-scraper

php-curl-class / php-curl-class
web-scraping,PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
Organization: php-curl-class
Home Page: https://www.phpcurlclass.com/
php curl class api api-client client framework http http-client http-proxy

platonai / pulsarrpa
web-scraping,Automate webpages at scale, scrape web data completely and accurately with high performance, distributed RPA.
User: platonai
web-crawler web-mining data-science web-sql crawler scraper scraping web-scraping data-mining rpa

postmodern / spidr
web-scraping,A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
User: postmodern
spider ruby spider-links crawler web scraper web-scraping web-spider web-crawler web-scraper

programminghistorian / jekyll
web-scraping,Jekyll-based static site for The Programming Historian
Organization: programminghistorian
Home Page: http://programminghistorian.org
programming-historian text-analysis api data-management data-manipulation data-mining pedagogy linked-open-data mapping network-analysis

roach-php / core
web-scraping,The complete web scraping toolkit for PHP.
Organization: roach-php
Home Page: https://roach-php.dev
php web-scraping crawling

roniemartinez / dude
web-scraping,dude uncomplicated data extraction: A simple framework for writing web scrapers using Python decorators
User: roniemartinez
Home Page: https://roniemartinez.github.io/dude/
python scraping framework playwright scraper xpath css web-scraping beautifulsoup4 parsel

rushter / selectolax
web-scraping,Python binding to Modest and Lexbor engines (fast HTML5 parser with CSS selectors).
User: rushter
css html5 modest-engine parser python web-scraping

sangaline / wayback-machine-scraper
web-scraping,A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.
User: sangaline
Home Page: http://sangaline.com/post/wayback-machine-scraper/
web-scraping wayback-machine wayback-archiver archive-dot-org python command-line-tool

scrapy / scrapy
web-scraping,Scrapy, a fast high-level web crawling & scraping framework for Python.
Organization: scrapy
Home Page: https://scrapy.org
crawler crawling framework hacktoberfest python scraping web-scraping web-scraping-python

seleniumbase / seleniumbase
web-scraping,📊 Python's all-in-one framework for web crawling, scraping, testing, and reporting. Supports pytest. UC Mode provides stealth. Includes many tools.
Organization: seleniumbase
Home Page: https://seleniumbase.io
python selenium webdriver selenium-python e2e-testing seleniumbase pytest-plugin web-automation pytest behave

serpapi / google-search-results-python
web-scraping,Google Search Results via SERP API pip Python Package
Organization: serpapi
python serp-api bing-image google-crawler google-images scraping serpapi web-scraping

shaikhsajid1111 / social-media-profile-scrapers
web-scraping,Fetch user's data across social media
User: shaikhsajid1111
web-scraping web-scraper request python selenium-python facebook-scraper twitter-scraper pinterest reddit-scraper medium-scraper

smartproxy / smartproxy
web-scraping,HTTP(S)/SOCKS5 rotating residential proxies - code examples & general information.
Organization: smartproxy
Home Page: https://smartproxy.com
proxy proxies https-proxy proxy-server python data-collection proxy-integration proxy-list python-scraper scraping

snooppr / snoop
web-scraping,Snoop — инструмент разведки на основе открытых данных (OSINT world)
User: snooppr
Home Page: https://github.com/snooppr/snoop/releases
blueteam ctf geo geocoder infosec ip nickname osint parser pentest police redteam scanner scraping security termux username username-checker username-search web-scraping

spekulatius / phpscraper
web-scraping,A universal web-util for PHP.
User: spekulatius
Home Page: https://phpscraper.de
php php-spider php-crawler scraping-websites php-scraper scraper scraping web-scraper web-scraping beautifulsoup

tidyverse / rvest
web-scraping,Simple web scraping for R
Organization: tidyverse
Home Page: https://rvest.tidyverse.org
r web-scraping html

vida-nyu / ache
web-scraping,ACHE is a web crawler for domain-specific search.
Organization: vida-nyu
Home Page: http://ache.readthedocs.io
domain-specific-search focused-crawler hacktoberfest web-crawler web-scraping web-search web-spider

vprusso / youtube_tutorials
web-scraping,Collection of scripts corresponding to LucidProgramming YouTube tutorials
User: vprusso
Home Page: http://bit.ly/lucidcode
python-tutorial web-scraping youtube-tutorial lucidprogramming ctci-solutions technical-interview python python3 python3-tutorial

x4nth055 / pythoncode-tutorials
web-scraping,The Python Code Tutorials
User: x4nth055
Home Page: https://www.thepythoncode.com
computer-vision ethical-hacking face-detection machine-learning natural-language-processing network-analysis network-programming network-security programming-tutorial python python-tutorials python3 scapy scapy-tutorials socket-programming text-classification tutorials web-scraping

yusuzech / r-web-scraping-cheat-sheet
web-scraping,Guide, reference and cheatsheet on web scraping using rvest, httr and Rselenium.
User: yusuzech
cheatsheet httr r rselenium rvest scrape-websites web-scraping webscraping

z0m31en7 / uscrapper
web-scraping,Uscrapper Vanta: Dive deeper into the web with this powerful open-source tool. Extract valuable insights with ease and efficiency, from both surface and deep web sources. Empower your data mining and analysis with Vanta's advanced capabilities. Fast, reliable, and user-friendly, Uscrapper Vanta is the ultimate choice for researchers and analysts.
User: z0m31en7
osint python web-scraping website-scraper information-extraction information-gathering osint-python osint-tool reconnaissance webscraping