Code Monkey home page Code Monkey logo

out0fmemory / guozhongcrawler Goto Github PK

View Code? Open in Web Editor NEW
97.0 12.0 66.0 29.33 MB

GuozhongCrawler的是一个无须配置、便于二次开发的爬虫开源框架，它提供简单灵活的API，只需少量代码即可实现一个爬虫。其设计灵感来源于多个爬虫国内外爬虫框架的总结。采用完全模块化的设计，功能覆盖整个爬虫的生命周期(链接提取、页面下载、内容抽取、持久化)，支持多线程抓取，分布式抓取，并支持自动重试，定制执行js、自定义cookie等功能。在处理网站抓取多次后被封IP的问题上，guozhongCrawler采用动态轮换IP机制有效防止IP被封。另外，源码中的注释及Log输出全部采用通俗易懂的中文。让初学者能有更加深刻的理解

Java 100.00%

guozhongcrawler's People

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

zeus911 caprice sayi21cn reed20150104 a707937337 anyfly8 tangzhezhi wangyx0055 zengjianxuan long19901225 yezhen eseawind augustlong silenyuan hpgh yongxu74 weieast sinomiko gloryhui troublrman eamonsec vaeaze hongye612430 jibaro qisw cuittzq smalltao crayfish07 eddylapis qingzeng ruoyingzhufeng xtuyaowu wozhidaomo kiss88 chaoshiang ttwwxk smx5555 songfang cc8848 loong2018 xiaodin1 xiaovc leejones92 kevinyzy1 ilukicc wxcmyx sniperxiaojun groupsell f361600134 zhaoguangyue1986 xiafobao hhy5277 mazhidong tsevae xiaoshan006 beifenku wuqd2048 zhangske dongtianqi1125 08xq chenkai1993 leoyang666 goroking juleyap88

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.