The youtube_livechat_replay_crawler from hase-ryo

continuation_livechat_crawlerでKeyerror 'shortcuts'が出て止まってしまうことがある

Issuesを初めて使うので作法に沿っていないコメントと思いますがご容赦ください。
convert_chatreplay内において、絵文字を含むチャットを取得した際、Keyerrorが出てそれ以降のチャットが取得できなくなることがあります。
エスケープ等で自力でどうにかできないか試してみましたが構文エラーになってしまいました。
以下に実行時のメッセージをいくつか丸ごとコピーします (Windowsのユーザー名のみhogeに差し替えています)。同じ動画を対象にした場合は何度繰り返しても必ず同じメッセージが出ます。
実行環境はWindows 10、Python 3.9.6、Git Bash version 2.33.0.windows.2です。

$ py -X utf8 locally_run.py OQYtJjOGWC4
https://www.youtube.com/watch?v=OQYtJjOGWC4
Page 16KeyError
'shortcuts'
{'message': {'runs': [{'text': '酔い止め求む'}, {'emoji': {'emojiId': '🆘', 'image': {'thumbnails': [{'url': 'https://www.youtube.com/s/gaming/emoji/7ff574f2/emoji_u1f198.svg'}]}}}]}, 'authorName': {'simpleText': 'よーぐるぐるっぺ'}, 'authorPhoto': {'thumbnails': [{'url': 'https://yt4.ggpht.com/ytc/AKedOLT1geJQEducxlDRiXCTolmEdGo3JJPMrQoiC8mjpi0=s32-c-k-c0x00ffffff-no-rj', 'width': 32, 'height': 32}, {'url': 'https://yt4.ggpht.com/ytc/AKedOLT1geJQEducxlDRiXCTolmEdGo3JJPMrQoiC8mjpi0=s64-c-k-c0x00ffffff-no-rj', 'width': 64, 'height': 64}]}, 'contextMenuEndpoint': {'commandMetadata': {'webCommandMetadata': {'ignoreNavigation': True}}, 'liveChatItemContextMenuEndpoint': {'params': 'Q2tjS1JRb2FRMHhsYzNCUFJ6Wnlaa2xEUm1WQlRuSlJXV1IzTnpSRFlVRVNKME5QTTFseFNqSTJjbVpKUTBaVVNGQlVRVWxrWDJ3MFFUaEJNVFl5T0Rnek9URTJPVEU1TlJvcEtpY0tHRlZEUkZnMVVHTkhaR2czTTJKdWNsWk5aMlJSWDBSdlFSSUxUMUZaZEVwcVQwZFhRelFnQVNnRU1ob0tHRlZEYm1aeWRuZHBZVXRqZVUxSlMzUXdZM0pVUWtodlFRJTNEJTNE'}}, 'id': 'CkUKGkNMZXNwT0c2cmZJQ0ZlQU5yUVlkdzc0Q2FBEidDTzNZcUoyNnJmSUNGVEhQVEFJZF9sNEE4QTE2Mjg4MzkxNjkxOTU%3D', 'timestampUsec': '1628839170610803', 'authorBadges': [{'liveChatAuthorBadgeRenderer': {'customThumbnail': {'thumbnails': [{'url': 'https://yt3.ggpht.com/7I4wi8BcCI1bxGqL13KHcGRCt9x6CaOGIkt-nrOSWHQkrboJ-jUvbq7k9WIAyXH-ZhplSsM1=s16-c-k'}, {'url': 'https://yt3.ggpht.com/7I4wi8BcCI1bxGqL13KHcGRCt9x6CaOGIkt-nrOSWHQkrboJ-jUvbq7k9WIAyXH-ZhplSsM1=s32-c-k'}]}, 'tooltip': 'メンバー（6 か月）', 'accessibility': {'accessibilityData': {'label': 'メンバー（6 か月）'}}}}], 'authorExternalChannelId': 'UCnfrvwiaKcyMIKt0crTBHoA', 'contextMenuAccessibility': {'accessibilityData': {'label': 'コメントの操作'}}, 'timestampText': {'simpleText': '5:26'}}
OQYtJjOGWC4 found 016 pages
DONE!

$ py -X utf8 locally_run.py H3LkVb0UlgI
https://www.youtube.com/watch?v=H3LkVb0UlgI
KeyError
'shortcuts'
Traceback (most recent call last):
  File "C:\Users\hoge\Desktop\python test\youtube_livechat_replay_crawler\continuation_livechat_crawler\..\continuation_livechat_crawler\main.py", line 118, in get_chat_replay_from_continuation
    chatlog = convert_chatreplay(item['liveChatPaidMessageRenderer'])
  File "C:\Users\hoge\Desktop\python test\youtube_livechat_replay_crawler\continuation_livechat_crawler\..\continuation_livechat_crawler\main.py", line 77, in convert_chatreplay
    content += runs['emoji']['shortcuts'][0]
KeyError: 'shortcuts'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\hoge\Desktop\python test\youtube_livechat_replay_crawler\continuation_livechat_crawler\locally_run.py", line 13, in <module>
    comment_data, continuation = get_chat_replay_from_continuation(video_id, continuation, 3000, True)
  File "C:\Users\hoge\Desktop\python test\youtube_livechat_replay_crawler\continuation_livechat_crawler\..\continuation_livechat_crawler\main.py", line 146, in get_chat_replay_from_continuation
    print(item['liveChatTextMessageRenderer'])
KeyError: 'liveChatTextMessageRenderer'

$ py -X utf8 locally_run.py an37cbt0o_E
https://www.youtube.com/watch?v=an37cbt0o_E
Page 4KeyError
'shortcuts'
{'message': {'runs': [{'text': 'こんばんはー'}, {'emoji': {'emojiId': '🙌🏻', 'image': {'thumbnails': [{'url': 'https://www.youtube.com/s/gaming/emoji/7ff574f2/emoji_u1f64c_1f3fb.svg'}]}}}]}, 'authorName': {'simpleText': 'ひよこレモン'}, 'authorPhoto': {'thumbnails': [{'url': 'https://yt4.ggpht.com/7aM-B_OgAeVqNzhwEUyt2ejbHLl2DuVY_jkyL4YjkZvWeQY_eiH5U2eUyJX2WLdt2Fpr9JUrFw=s32-c-k-c0x00ffffff-no-rj', 'width': 32, 'height': 32}, {'url': 'https://yt4.ggpht.com/7aM-B_OgAeVqNzhwEUyt2ejbHLl2DuVY_jkyL4YjkZvWeQY_eiH5U2eUyJX2WLdt2Fpr9JUrFw=s64-c-k-c0x00ffffff-no-rj', 'width': 64, 'height': 64}]}, 'contextMenuEndpoint': {'commandMetadata': {'webCommandMetadata': {'ignoreNavigation': True}}, 'liveChatItemContextMenuEndpoint': {'params': 'Q2p3S09nb2FRMHBwYWpkdlUxRjZVRWxEUm1OVlRuSlJXV1F6U1RCT2VWRVNIRU5MVTNwZlgzRlFlbEJKUTBaU1NHdHFkMjlrUlZkWlJEUjNMVEFhS1NvbkNoaFZRMFJZTlZCalIyUm9Oek5pYm5KV1RXZGtVVjlFYjBFU0MyRnVNemRqWW5Rd2IxOUZJQUVvQkRJYUNoaFZRekpHY0hkeFExRk9URUZoWjNGc1gycHFXV2xqVm5jJT
```NE'}}, 'id': 'CjoKGkNKaWo3b1NRelBJQ0ZjVU5yUVlkM0kwTnlREhxDS1N6X19xUHpQSUNGUkhrandvZEVXWUQ0dy0w', 'timestampUsec': '1629892854387139', 'authorBadges': [{'liveChatAuthorBadgeRenderer': {'customThumbnail': {'thumbnails': [{'url': 'https://yt3.ggpht.com/nyHIwHuC808mPLic9OQk-g25kvAN0h13IoXDsWUU9LwSzc6c88SrvzT3bhZjyx5fowiFz_lBmfI=s16-c-k'}, {'url': 'https://yt3.ggpht.com/nyHIwHuC808mPLic9OQk-g25kvAN0h13IoXDsWUU9LwSzc6c88SrvzT3bhZjyx5fowiFz_lBmfI=s32-c-k'}]}, 'tooltip': 'メンバー（1 か月）', 'accessibility': {'accessibilityData': {'label': 'メンバー（1 か月）'}}}}], 'authorExternalChannelId': 'UC2FpwqCQNLAagql_jjYicVw', 'contextMenuAccessibility': {'accessibilityData': {'label': 'コメントの操作'}}, 'timestampText': {'simpleText': '1:50'}}
an37cbt0o_E found 004 pages
DONE!

hase-ryo / youtube_livechat_replay_crawler Goto Github PK

youtube_livechat_replay_crawler's People

Contributors

Stargazers

Watchers

Forkers

youtube_livechat_replay_crawler's Issues

continuation_livechat_crawlerでKeyerror 'shortcuts'が出て止まってしまうことがある

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent