Code Monkey home page Code Monkey logo

Comments (16)

thekingofcity avatar thekingofcity commented on June 7, 2024

自行检查是否任务启动及流程

from weibospider.

myrainbowandsky avatar myrainbowandsky commented on June 7, 2024

怎么启动,

自行检查是否任务启动及流程

具体哪个命令启动

from weibospider.

thekingofcity avatar thekingofcity commented on June 7, 2024

fans_followers这个队列

from weibospider.

OneCodeMonkey avatar OneCodeMonkey commented on June 7, 2024

没有抓过粉丝列表和关注列表。这个表就是空的

from weibospider.

myrainbowandsky avatar myrainbowandsky commented on June 7, 2024

没有抓过粉丝列表和关注列表。这个表就是空的

截屏2020-03-2116 49 15

报错

from weibospider.

myrainbowandsky avatar myrainbowandsky commented on June 7, 2024

fans_followers这个队列

报错

[2020-03-21 16:54:57,151: INFO/MainProcess] Received task: tasks.user.crawl_follower_fans[2768cfb2-337d-4cbf-8aea-635c971d45de]  
[2020-03-21 16:54:57,153: ERROR/ForkPoolWorker-1] Task tasks.user.crawl_follower_fans[6c223b6f-8f48-46b3-b59d-c308763835fb] raised unexpected: AttributeError("'NoneType' object has no attribute 'other_crawled'",)
Traceback (most recent call last):
  File "/home/wentao/programming/weibospider/WeiboSpider/lib/python3.6/site-packages/celery/app/trace.py", line 382, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/wentao/programming/weibospider/WeiboSpider/lib/python3.6/site-packages/celery/app/trace.py", line 641, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/wentao/programming/weibospider/tasks/user.py", line 12, in crawl_follower_fans
    if seed.other_crawled == 0:
AttributeError: 'NoneType' object has no attribute 'other_crawled'

from weibospider.

thekingofcity avatar thekingofcity commented on June 7, 2024

检查表seed_ids

from weibospider.

myrainbowandsky avatar myrainbowandsky commented on June 7, 2024

检查表seed_ids

seed_ids 也里有东西,:

截屏2020-03-2212 38 04

但是以下三个表一直空
截屏2020-03-2212 39 46

from weibospider.

thekingofcity avatar thekingofcity commented on June 7, 2024

检查crawl_follower_fans函数入参以及seed_ids是否有此条uid记录

from weibospider.

myrainbowandsky avatar myrainbowandsky commented on June 7, 2024

检查crawl_follower_fans函数入参以及seed_ids是否有此条uid记录

在user.py打印uid 是可以的:
截屏2020-03-2314 25 34

但是提示seed没有other_crawled属性

截屏2020-03-2314 23 32

在dao.py找不到other_crawled属性
截屏2020-03-2314 31 30

from weibospider.

thekingofcity avatar thekingofcity commented on June 7, 2024

那大概便是```seed_ids``无此条uid记录

from weibospider.

myrainbowandsky avatar myrainbowandsky commented on June 7, 2024

那大概便是```seed_ids``无此条uid记录

crawl_follower_fans 是从seed_ids 读取数据吗?

seed_ids现在有大批数据,怎么让 crawl_follower_fans 读呢
Screenshot from 2020-03-23 20-25-45

from weibospider.

thekingofcity avatar thekingofcity commented on June 7, 2024

可以尝试重写11行SeedidsOper.get_seed_by_id(uid)
使用类似这样的函数SeedidsOper.get_seed_ids从数据库中获取数据

from weibospider.

myrainbowandsky avatar myrainbowandsky commented on June 7, 2024

可以尝试重写11行SeedidsOper.get_seed_by_id(uid)
使用类似这样的函数SeedidsOper.get_seed_ids从数据库中获取数据

我先去试一试看看。

from weibospider.

myrainbowandsky avatar myrainbowandsky commented on June 7, 2024

可以尝试重写11行SeedidsOper.get_seed_by_id(uid)
使用类似这样的函数SeedidsOper.get_seed_ids从数据库中获取数据

报错,是一个bug吗?

[2020-03-25 19:07:12,388: ERROR/ForkPoolWorker-1] Task tasks.user.crawl_follower_fans[23f3c1fd-fc6e-4c5b-b0cc-5d5c6a9ad068] raised unexpected: TypeError('expected string or bytes-like object',)
Traceback (most recent call last):
  File "/home/wentao/programming/weibospider/WeiboSpider/lib/python3.6/site-packages/celery/app/trace.py", line 382, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/wentao/programming/weibospider/WeiboSpider/lib/python3.6/site-packages/celery/app/trace.py", line 641, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/wentao/programming/weibospider/tasks/user.py", line 19, in crawl_follower_fans
    rs = get_fans_or_followers_ids(uid, 1, 1)
  File "/home/wentao/programming/weibospider/page_get/user.py", line 159, in get_fans_or_followers_ids
    urls_length = public.get_max_crawl_pages(page)
  File "/home/wentao/programming/weibospider/page_parse/user/public.py", line 223, in get_max_crawl_pages
    m = re.search(pattern, script.string)
  File "/usr/lib/python3.6/re.py", line 182, in search
    return _compile(pattern, flags).search(string)
TypeError: expected string or bytes-like object

user.py 中 script.string 有bug
script.string 这里有bug无法判断是哪个类型一会nontype,一会是<class 'bs4.element.NavigableString'>
无论哪种类型都无法用re 模块抓取

 for script in scripts:
        #print('i am in '+dir_path,'script is '+script)
        #print('script.string:',script.string)
        
        print('type pattern',pattern)
        print('pattern', pattern)
        print('type:',type(script.string))
        
        m = re.search(pattern, script.string)


        if m and 'pl.content.followTab.index' in script.string:
            all_info = m.group(1)
            cont = json.loads(all_info).get('html', '')
            soup = BeautifulSoup(cont, 'html.parser')
            pattern = 'uid=(.*?)&'

            if 'pageList' in cont:
                urls2 = soup.find(attrs={'node-type': 'pageList'}).find_all(attrs={
                    'class': 'page S_txt1', 'bpfilter': 'page'})
                length += len(urls2)
    return length

截屏2020-03-2521 14 25

from weibospider.

thekingofcity avatar thekingofcity commented on June 7, 2024

与issue无关, 建议新开issue

from weibospider.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.