hezhang1994 / weibo-crawler Goto Github PK

It seems I can only find "getdashinfoxxx" and "proxy?xxx" on my XHR tag. Is there other ways to find the containder id? Thankss

It happens at the following code:
if card['mblog']['isLongText'] == 'False': ... else: ...

In details, when card['mblog']['isLongText'] = False, the code still excutes the else content.

因为某些微博表情无法解码保存。

see "bmiddle_pic" under "retweeted_status" and modify the code.

1. If the cookie is expired, the crawling process will stop.

   • How to continue to crawl data from the stopped post?
   • How to automatic crawl data?

2. How to crawl video?

换了不同人的账号，一到live photo就会报错
FileNotFoundError: [WinError 2] 系统找不到指定的文件
谢谢

Error message:
Traceback (most recent call last):
File "H:/prbkp/weibo-crawler/run_WeiboCrawler.py", line 245, in
photo_str = card['mblog']['pic_video']
KeyError: 'pic_video'

I do not know why it appeared. I just use:
if card_info:
if 'pic_video' in card_info:
try:
photo_str = card['mblog']['pic_video']
photo_list = re.split('[,]', photo_str)
for photo in photo_list:
# E.g., 'photo' = '0:000voDMsjx07t57qM583010f0100alDF0k01'.
photo_code = re.split('[:]', photo)[1]
photo_url = photo_url_start + photo_code + photo_url_end
photo_urls.append(photo_url)
except:
pass

to skip the part of code to prevent sudden interruption.

Hope it can help for further improvement.

1. If the cookie is expired, the crawling process will stop.

   • How to continue to crawl data from the stopped post?
   • How to automatic crawl data?

2. How to crawl videos?

3. How to crawl GIFs?

It seems like this code cannot crawl video from re-tweeted post.

Currently, this implementation can not crawl images posted in transferred text.

Sometimes, the value of time_list could be ['N/A\r'] (def time_str2float). It may mean the video missed. Then, it would break.
My temporary solution is that set 'time_num = 0', if the 'N/A\r' appears. However, I do not know if it would cause unknown flaws.