Code Monkey home page Code Monkey logo

Comments (6)

ResolveWang avatar ResolveWang commented on May 17, 2024

可能出现的问题就是账号被封,其它没啥问题,你要账号多的话,可以频繁一些。而且抓取不同模块,微博的不同模块限制的抓取间隔也不一样,目前还没做细粒度的控制。

from weibospider.

lovenodejs avatar lovenodejs commented on May 17, 2024

最想用的是用户主页的抓取,和搜索模块,如果是一个账号推荐最低的间隔是多少,

from weibospider.

ResolveWang avatar ResolveWang commented on May 17, 2024

搜索的话,限制很严格,如果一个账号的话,你试试20~30秒吧,用户主页的话,大概十来秒。这个我也不能说绝对是这样,只能给一个大概的范围

from weibospider.

lovenodejs avatar lovenodejs commented on May 17, 2024

那岂不是在源码文件中原来设置的小时已经是很宽松了的?也就是在tasks/workers.py 设置 hours minutes seconds 成需要间隔多少时间对应的值是吗?

from weibospider.

ResolveWang avatar ResolveWang commented on May 17, 2024

读文档吧。我感觉你没读文档,我希望你是读了文档再来提问的。

一个是抓取间隔,一个是定时任务间隔,两个是不一样的

spider.yaml中是抓取间隔,就是每两个http request的发送间隔

min_crawl_interal: 10 # min interal of http request
max_crawl_interal: 20 # max interal of http request

workers.py中是定时任务间隔,比如你要用微博搜索,我会在这一轮搜完过后,再过N个小时再进行搜索,而不会这轮搜完马上就进行下一轮搜索

from weibospider.

lovenodejs avatar lovenodejs commented on May 17, 2024

好的,麻烦了

from weibospider.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.