pyscrape Goto Github PK

python-patterns

A collection of design patterns/idioms in Python

python-regex-cheatsheet

Python 2.7 Regular Expression cheatsheet, as a restructured text document and Makefile to convert it to PDF

python-twitter

A Python wrapper around the Twitter API.

pythonspidernotes

Python入门网络爬虫之精华版

requests

Python HTTP Requests for Humans™.

routersploit

Exploitation Framework for Embedded Devices

scapy

Scapy: the Python-based interactive packet manipulation program & library. Supports Python 2 & Python 3.

scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

scrapy-deltafetch

Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls

scrapy-dynamic-configurable

A dynamic configurable news crawler based Scrapy

scrapy-elasticsearch

A scrapy pipeline which send items to Elastic Search server

scrapy-examples

Multifarious Scrapy examples. Spiders for alexa / amazon / douban / douyu / github / linkedin etc.

scrapy-fake-useragent

Random User-Agent middleware based on fake-useragent

scrapy-graphite

Output scrapy statistics to graphite/carbon

scrapy-kafka

Kafka-based components for Scrapy

scrapy-rabbitmq

A RabbitMQ Scheduler for Scrapy

scrapy-selenium

Scrapy middleware to handle javascript pages using selenium

scrapy-spiders

Collection of python scripts I have created to crawl various websites, mostly for lead generation projects to match keywords and collect email addresses and post URLs

scrapy-wayback-machine

A Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

scrapy_tripadvisor_restaurant

I wrote the script to scrape the restaurant information in Las Vegas.

scrapyai

scrapyd

A service daemon to run Scrapy spiders

scrapylib

Collection of Scrapy utilities (extensions, middlewares, pipelines, etc)

scrubadub

Fork of the PII removal tool:

security_monkey

Security Monkey monitors AWS, GCP, OpenStack, and GitHub orgs for assets and their changes over time.

service-domain-checker

A web application that checks whether transactions on GOV.UK meet the recommendations of the Service Manual

social-media-and-contact-info-extractor

Run this scraper for free: https://apify.com/vdrmota/contact-info-scraper

spider-code

scrapy分布式爬虫，selenium 爬虫，手机群控（自动化）（appium,airtest,uiautomator2），反爬破解文档

spiderfoot

SpiderFoot automates OSINT collection so that you can focus on analysis.

sqlmap

Automatic SQL injection and database takeover tool