when time_list == 'N/A\r', it would break about weibo-crawler HOT 4 CLOSED

Hi eweca1992,

Thanks for pointing out this issue!

I tried to set "time_list = 'NAr'" in "def time_str2float", and got "ValueError: invalid literal for int() with base 10" (I would like to suggest you check the name of the error you have met, here is "ValueError").

Thus, I modified this function by adding "try" and "except" operations as below:

def time_str2float2(timestamp):
    '''Convert a timestamp to a floating number.
    Parameter:
        timestamp <str> - The input timestamp formatted as 'hh:mm:ss.msec'.
    Return:
        time_num <float> - The number of the input timestamp formatted as 'ss.msec'.
    '''
    time_list = re.split('[:]', timestamp)
    time_list = 'N/A\r'
    try:
        time_num = int(time_list[0]) * 3600 + int(time_list[1]) * 60 + float(time_list[2])
        return time_num
    except ValueError:
        print('*****Error: Invalid format of timestamp')
        return 0

You could replace "ValueError" in "except ValueError" with the name of error you have met.

I hope this could help you to fix your issue. :P

Regards,
He

from weibo-crawler.

Hi eweca1992,

Thanks for pointing out this issue!

I tried to set "time_list = 'NAr'" in "def time_str2float", and got "ValueError: invalid literal for int() with base 10" (I would like to suggest you check the name of the error you have met, here is "ValueError").

Thus, I modified this function by adding "try" and "except" operations as below:

def time_str2float2(timestamp):
    '''Convert a timestamp to a floating number.
    Parameter:
        timestamp <str> - The input timestamp formatted as 'hh:mm:ss.msec'.
    Return:
        time_num <float> - The number of the input timestamp formatted as 'ss.msec'.
    '''
    time_list = re.split('[:]', timestamp)
    time_list = 'N/A\r'
    try:
        time_num = int(time_list[0]) * 3600 + int(time_list[1]) * 60 + float(time_list[2])
        return time_num
    except ValueError:
        print('*****Error: Invalid format of timestamp')
        return 0

You could replace "ValueError" in "except ValueError" with the name of error you have met.

I hope this could help you to fix your issue. :P

Regards,
He

Thank you for your reply. I would try it when my computer is free.

By the way, is it possible to auto re-connect weibo after request error? It seems that a "try...except" can do the work.

from weibo-crawler.

Hi eweca1992,
Thanks for pointing out this issue!
I tried to set "time_list = 'NAr'" in "def time_str2float", and got "ValueError: invalid literal for int() with base 10" (I would like to suggest you check the name of the error you have met, here is "ValueError").
Thus, I modified this function by adding "try" and "except" operations as below:

def time_str2float2(timestamp):
    '''Convert a timestamp to a floating number.
    Parameter:
        timestamp <str> - The input timestamp formatted as 'hh:mm:ss.msec'.
    Return:
        time_num <float> - The number of the input timestamp formatted as 'ss.msec'.
    '''
    time_list = re.split('[:]', timestamp)
    time_list = 'N/A\r'
    try:
        time_num = int(time_list[0]) * 3600 + int(time_list[1]) * 60 + float(time_list[2])
        return time_num
    except ValueError:
        print('*****Error: Invalid format of timestamp')
        return 0

You could replace "ValueError" in "except ValueError" with the name of error you have met.
I hope this could help you to fix your issue. :P
Regards,
He

Thank you for your reply. I would try it when my computer is free.

By the way, is it possible to auto re-connect weibo after request error? It seems that a "try...except" can do the work.

Hi,

Yes! First of all, the code will try at most 3 times to crawl the data of one Weibo post (3 "try-except" code in "def crawl_***") if "urllib.request.urlretrieve" function failed to download the data. If the code still cannot crawl the data after 3 times attempts, the data will not be saved. (I have never met this problem so far.)

Second, if you get the error: "ConnectionError 104: ('Connection aborted.')" (the most common one which is caused by the restriction of Sina Weibo. It is different with the first problem), the code will stop! In this case, you could follow the information described in the 8-th procedure of User Settings of Usage in README and manually re-start the code to continue to crawl data. Currently, the crawler cannot automatically do this. I am sorry. XD

Regards,
He

from weibo-crawler.

Thank you for your patient reply.

from weibo-crawler.