Code Monkey home page Code Monkey logo

spider's Introduction

Spider爬虫实例

1. dianping——大众点评

  • 大众点评获取所有排行榜店铺信息建表入库
  • selenium登录获取cookie,使用cookie模拟登录
  • 从库中获取所有店铺的id,拼接URL获取每个店铺所有评论信息并建表入库
  • 添加断点续爬功能,程序中断后自动从断点开始爬取。

2. zhihuCrawl——知乎web

  • 知乎模拟登录
  • 验证码图片保存,输入识别的验证码
  • 获取当日热门话题及其高赞回答
  • 保存问题标题、链接,回答者的ID,内容及点赞数量,存储至MongoDB

3. 微信公众号爬虫

  • 使用西刺代理构建代理ip池,检查代理ip连通性
  • 使用可用的ip爬取搜狗微信公众号文章
  • 将文章保存为.doc文档

4. 豆瓣读书爬虫

  • 读取豆瓣每个分类的书籍列表
  • 获取每本书信息,存储到MySQL,爬取完毕后删除重复数据并重新排序

5. csdn_scrapy——csdn网站

  • 爬取csdn所有子标签文章,清洗后保存到MongoDB
  • 使用scrapy.redis组件进行分布式部署

6. bilibili——b站

  • 使用scrapy-splash爬取b站子标签热门video排行的大量信息
  • 按标签建表保存至MongoDB

7. toutiao——今日头条web

  • 爬取今日头条子标签新闻
  • webdriver获取加密参数
  • ajax请求返回json数据解析,保存至MongoDB

8. taobao——淘宝web

  • 爬取淘宝各子标签,按销量排名商品信息,按分类保存至MongoDB
  • 通过pandas进行数据分析
  • 将商品在各省分布、销量排行、地图分布等通过matplotlib绘图显示

9. zhihu_app——知乎app

  • 爬取使用ssl-pinning技术的app
  • fiddler抓包获取api请求
  • 模拟手机端请求数据,解析json存入mysql

10. Ajax_weibo——微博app

  • 输入要爬取的博主ID,获取ajax请求
  • 解析json数据,爬取博主所有微博,保存至MySQL

11. scrapyd

docker scrapyd 配置文件

spider's People

Contributors

malei666 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

spider's Issues

可不可以爬爬各大主流新闻网站新闻链接

   正在用flutter尝试写一个新闻APP,需要一些新闻链接,不需要内容清洗,直接链接跳转到对应网站上的文章,免得有版权问题,

  APP类似“今日热榜”

 看过RSSHub,也提供一些新闻链接,不过太少了,

作者您这儿会不会有相应的计划?

知乎登录403

知乎登录时返回403错误码,登录好像做得更复杂了

connection was refused?

$ scrapy crawl tt
2018-10-19 15:01:15 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: toutiao)
2018-10-19 15:01:15 [scrapy.utils.log] INFO: Versions: lxml 4.2.3.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.0, w3lib 1.19.0, Twisted 18.7.0, Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 05:52:31) - [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)], pyOpenSSL 18.0.0 (OpenSSL 1.1.0h 27 Mar 2018), cryptography 2.3, Platform Darwin-18.0.0-x86_64-i386-64bit
2018-10-19 15:01:15 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'toutiao', 'COOKIES_ENABLED': False, 'DOWNLOAD_DELAY': 3, 'DUPEFILTER_CLASS': 'scrapy_splash.SplashAwareDupeFilter', 'HTTPCACHE_STORAGE': 'scrapy_splash.SplashAwareFSCacheStorage', 'NEWSPIDER_MODULE': 'toutiao.spiders', 'REDIRECT_ENABLED': False, 'SPIDER_MODULES': ['toutiao.spiders']}
2018-10-19 15:01:15 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.logstats.LogStats']
2018-10-19 15:01:15 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy_splash.SplashCookiesMiddleware',
'scrapy_splash.SplashMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-10-19 15:01:15 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy_splash.SplashDeduplicateArgsMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-10-19 15:01:15 [scrapy.middleware] INFO: Enabled item pipelines:
['toutiao.pipelines.ToutiaoPipeline']
2018-10-19 15:01:15 [scrapy.core.engine] INFO: Spider opened
2018-10-19 15:01:15 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-10-19 15:01:15 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-10-19 15:01:16 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.toutiao.com via http://localhost:8050/render.html> (failed 1 times): Connection was refused by other side: 61: Connection refused.
2018-10-19 15:01:19 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.toutiao.com via http://localhost:8050/render.html> (failed 2 times): Connection was refused by other side: 61: Connection refused.
2018-10-19 15:01:23 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET https://www.toutiao.com via http://localhost:8050/render.html> (failed 3 times): Connection was refused by other side: 61: Connection refused.
2018-10-19 15:01:23 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.toutiao.com via http://localhost:8050/render.html>
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
defer.returnValue((yield download_func(request=request,spider=spider)))
twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 61: Connection refused.
2018-10-19 15:01:23 [scrapy.core.engine] INFO: Closing spider (finished)
2018-10-19 15:01:23 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 3,
'downloader/exception_type_count/twisted.internet.error.ConnectionRefusedError': 3,
'downloader/request_bytes': 1818,
'downloader/request_count': 3,
'downloader/request_method_count/POST': 3,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2018, 10, 19, 7, 1, 23, 667284),
'log_count/DEBUG': 4,
'log_count/ERROR': 1,
'log_count/INFO': 7,
'memusage/max': 58916864,
'memusage/startup': 58912768,
'retry/count': 2,
'retry/max_reached': 1,
'retry/reason_count/twisted.internet.error.ConnectionRefusedError': 2,
'scheduler/dequeued': 4,
'scheduler/dequeued/memory': 4,
'scheduler/enqueued': 4,
'scheduler/enqueued/memory': 4,
'splash/render.html/request_count': 1,
'start_time': datetime.datetime(2018, 10, 19, 7, 1, 15, 822066)}
2018-10-19 15:01:23 [scrapy.core.engine] INFO: Spider closed (finished)

这个是为什么呀

zhihu 爬虫

可以和您交流一下知乎爬虫的相关问题嘛

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.