notatallshaw / arandabot Goto Github PK

either push in to the clipboard or write to a file the user can open. Or use the functionality to open the browser directly if you can get that to work properly.

Currently assumption is channel name or id will always exist, this could cause bot to get caught in long loop trying resolve non-existent name

As per this SO answer, it is apparently much faster to use the search functionality than to use playlist lookup: http://stackoverflow.com/questions/29204271/are-upload-playlists-on-youtube-api-v3-purpsofully-slow-to-be-updated/29441811#29441811

Can use activities list and set home = true:
https://developers.google.com/youtube/v3/docs/activities/list

I don't 100% trust this will be updated with all channel content but it might be worth merging this feed in to the requests in case it's more performant in general?

Once added logging add option to keep updating to show script is still working.

A common cient_secrets.json will be kept in the git repository. It will be used unless there exists a user cient_secret*.json in the bot directory.

The code will ignore provided settings when using common client_secrets.json, it will limit caling youtube to once every 60 seconds. There will be no obfuscation, if some user chooses to break it, that is up to them.

https://developers.google.com/youtube/v3/guides/push_notifications

Allow API v2 (if it really isn't decommed), APIv3, RSS Feeds, to all run in parallel to get data fastest possible way

Add lots of unit tests to help stop regressions.

Traceback (most recent call last):
File "main.py", line 19, in
main()
File "main.py", line 16, in main
arandabot.arandabot(settings=settings)
File "C:\Users\Damian\Documents\arandabot - prod\arandabot.py", line 46, in arandabot
yt.getNewestVideos()
File "C:\Users\Damian\Documents\arandabot - prod\ytvideos.py", line 246, in getNewestVideos
self.youtube = self.initilize_youtube(self.set)
File "C:\Users\Damian\Documents\arandabot - prod\ytvideos.py", line 119, in initilize_youtube
http=credentials.authorize(httplib2.Http()))
File "C:\Python27\lib\site-packages\oauth2client\util.py", line 132, in positional_wrapper
return wrapped(_args, *kwargs)
File "C:\Python27\lib\site-packages\apiclient\discovery.py", line 192, in build
resp, content = http.request(requested_url)
File "C:\Python27\lib\site-packages\oauth2client\util.py", line 132, in positional_wrapper
return wrapped(args, *kwargs)
File "C:\Python27\lib\site-packages\oauth2client\client.py", line 490, in new_request
redirections, connection_type)
File "C:\Python27\lib\site-packages\httplib2__init.py", line 1593, in request
(response, content) = self.request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "C:\Python27\lib\site-packages\httplib2__init.py", line 1335, in request
(response, content) = self.conn_request(conn, request_uri, method, body, headers)
File "C:\Python27\lib\site-packages\httplib2__init.py", line 1291, in _conn_request
response = conn.getresponse()
File "C:\Python27\lib\httplib.py", line 1061, in getresponse
raise ResponseNotReady()
httplib.ResponseNotReady

Use the new API v3 method: https://developers.google.com/youtube/v3/docs/commentThreads

Add in to the settings

Make it much easier to set up for the first time by adding an interactive walk through script, allowing user to put in username and password and channel configuration details of for the script to handle the saving to json

Enable universal flag for python

where the settings file is, where the client_secrets file is where to write the oauth file to etc...

Add more output capabilities, twitter?

Low Priority

I've been running the bot for a while and come across all these exceptions

This would add more resiliency as it seems less flaky than oauth

And make sure it's using the newest version of the code

Password based authentication is being removed:
https://www.reddit.com/comments/2ujhkr/

Currently got an error like this and not sure where it came from:

Some unexpected exception occured in arandabot backing off for 5 mins and trying
again:
'charmap' codec can't encode character u'\u2019' in position 38: character maps
to

o see if there is a file named in the style that you download from Google and if there's just 1 try that.

config goes somewhere in settings

Affects both main and search branch:

Traceback (most recent call last):
File "main.py", line 19, in
main()
File "main.py", line 16, in main
arandabot.arandabot(settings=settings)
File "C:\Users\Damian\Documents\arandabot - prod - search\arandabot.py", line46, in arandabot
yt.getNewestVideos()
File "C:\Users\Damian\Documents\arandabot - prod - search\ytvideos.py", line 251, in getNewestVideos
self.youtube = self.initilize_youtube(self.set)
File "C:\Users\Damian\Documents\arandabot - prod - search\ytvideos.py", line 120, in initilize_youtube
http=credentials.authorize(httplib2.Http()))
File "C:\Python27\lib\site-packages\oauth2client\util.py", line 132, in positional_wrapper
return wrapped(_args, *kwargs)
File "C:\Python27\lib\site-packages\apiclient\discovery.py", line 192, in build
resp, content = http.request(requested_url)
File "C:\Python27\lib\site-packages\oauth2client\util.py", line 132, in positional_wrapper
return wrapped(args, *kwargs)
File "C:\Python27\lib\site-packages\oauth2client\client.py", line 490, in new_request
redirections, connection_type)
File "C:\Python27\lib\site-packages\httplib2__init.py", line 1593, in request
(response, content) = self.request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "C:\Python27\lib\site-packages\httplib2__init.py", line 1335, in request
(response, content) = self.conn_request(conn, request_uri, method, body, headers)
File "C:\Python27\lib\site-packages\httplib2__init.py", line 1263, in _conn_request
raise ServerNotFoundError("Unable to find the server at %s" % conn.host)
httplib2.ServerNotFoundError: Unable to find the server at www.googleapis.com

Noticed an issue where script just isn't pulling some subscribed videos when it's looping for a long time, need to investigate

Once the number of subscription channel requests has got very large it is very slow to loop around getting all the user upload playlists. Should add option to multi-thread this step.

lots of logging

A users subscriptions may be updated while the script is running. Should allow to check if this has happened.

Message looks like this:
<HttpError 403 when requesting https://www.googleapis.com/youtube/v3/search?channelId=UCBa659QWEk1AI4Tg--mrJ2A&maxResults=50&safeSearch=none&part=snippet&alt=json&type=video&order=date returned "Quota Exceeded">

Back off for some time when this happens, print to screen why backing off for some time

Will need to depreciate password login to reddit in favor of ouath 2.0

It's a little unreliable, instead keep a list of videos IDs already posted.

Things to consider:

1. How big is the list allowed to get?
2. Is the list saved to file?
3. How do we use it to determine if we no longer need to search older videos? (I was thinking of having a confidence value, e.g. if more than 50% of videos on this search are already in our list, no need to check out older videos)

This will be used to populate all the default values and be kept in the git repository. If settings.json is not in the directory of the bot it will create it based of settings.tempate.json and user input from various questions.

settings.json will be added to the git ignore file so people's secret details (such as reddit username/password) aren't accidentally uploaded)

Add a post:true post:false for each channel in the JSON, allowing not to just manually add channels but to blacklist certain channels from the subscription list.

Some channels don't have user names, e.g. https://www.youtube.com/channel/UCaVwpbqM8dkhQvbL8XileAA/featured

I think sometimes the Google API classes just stop responding as apposed to throwing an exception. The idea to this would to be put the calls in to a separate thread / subprocess and kill the thread after a set amount of time if nothing happens.

Maybe measuring "nothing happens" could be via a queue which is populated by the loops in the method.

Currently dependencies are noted in the readme.md of this project, the more 'pythonic' and maintainable way to do this would be to pipe the requirements into a requirements.txt file.

You generate this by running the following command:
pip freeze >requirements.txt

commit the requirements file to your repo.

Then anyone can install all your dependencies by running:
pip install -r requirements.txt

Add a timer or an input capture to allow user to read instructions