xoelop / get-twitter-likes Goto Github PK

View Code? Open in Web Editor NEW

41.0 2.0 0.0 549 KB

Keeping a searchable collection of your Twitter likes

License: GNU General Public License v3.0

Python 6.27% JavaScript 7.02% Jupyter Notebook 86.28% Shell 0.44%

get-twitter-likes's Introduction

Check out this post for very detailed instructions on how to set everything up

get-twitter-likes's People

Contributors

Stargazers

Watchers

get-twitter-likes's Issues

Confirm Twitter API Keys?

Cool project! Got the whole thing set up and pulled my first 4000 tweet likes back. Only when I viewed them in Tinybird I had never seen any of the tweets before and half of them are in a language I don't speak 😂

Can you confirm that the Twitter API keys/secrets in the .env should come from this location on developer.twitter.com?

TWITTER_API_KEY=...
TWITTER_API_SECRET=...
TWITTER_ACCESS_TOKEN=...
TWITTER_ACCESS_TOKEN_SECRET=...

If so, I have no idea whose tweets I'm looking at.

Can't find pandas on Fedora 32

The following is shown when running get-twitter-likes:
Creating a virtualenv for this project…
Pipfile: /home/pyz/projects/Documents/Twitter/get-twitter-likes-master/Pipfile
Using /usr/bin/pypy3.6 (3.6.9) to create virtualenv…
⠦ Creating virtual environment...Already using interpreter /usr/bin/pypy3.6
Using base prefix '/usr/lib64/pypy3-7.3'
New pypy executable in /home/pyz/.local/share/virtualenvs/get-twitter-likes-master-VT8kSR22/bin/pypy3.6
Also creating executable in /home/pyz/.local/share/virtualenvs/get-twitter-likes-master-VT8kSR22/bin/pypy
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /usr/bin/pypy3.6

✔ Successfully created virtual environment!
Virtualenv location: /home/pyz/.local/share/virtualenvs/get-twitter-likes-master-VT8kSR22
Loading .env environment variables…
Traceback (most recent call last):
File "get_likes_to_csv.py", line 1, in
from src.core import get_all_statuses, create_df_statuses
File "/home/pyz/projects/Documents/Twitter/get-twitter-likes-master/src/core.py", line 10, in
import pandas as pd
ModuleNotFoundError: No module named 'pandas'

But pandas is installed:
rpm -q python3-pandas
python3-pandas-0.25.3-2.fc32.x86_64

Does the script need an absolute path to the requested modules?

sqlite3.OperationError: database is locked

Ok, I've got the daily stuff up and running and linked to a spreadsheet. Cool stuff!

Now moving on to adding likes from the like.js in the twitter backup file. It runs for about 90 seconds, creates a ~1GB cache.sqlite file, and then errors out with the following message:

(get-twitter-likes) user1@jh-ltm2 get-twitter-likes % python get_likes_from_json_to_csv.py -pu -o data/likes1.csv
Namespace(input='data/like.js', output='data/likes1.csv', format='raw', parse_urls=True, save_json_col=False)
Reading likes from /Users/user1/Projects/get-twitter-likes/data/like.js
Downloading likes detailed data from Twitter API and parsing their URLs
Traceback (most recent call last):                                                                                               
  File "/Users/user1/Projects/get-twitter-likes/get_likes_from_json_to_csv.py", line 28, in <module>
    likes = get_likes_from_json(input_file=input_file,
  File "/Users/user1/Projects/get-twitter-likes/src/core.py", line 166, in get_likes_from_json
    tweets_lists = parallel(lookup_and_parse_tweets,
  File "/Users/user1/.local/share/virtualenvs/get-twitter-likes-b1tRvag8/lib/python3.9/site-packages/fastcore/parallel.py", line 106, in parallel
    return L(r)
  File "/Users/user1/.local/share/virtualenvs/get-twitter-likes-b1tRvag8/lib/python3.9/site-packages/fastcore/foundation.py", line 97, in __call__
    return super().__call__(x, *args, **kwargs)
  File "/Users/user1/.local/share/virtualenvs/get-twitter-likes-b1tRvag8/lib/python3.9/site-packages/fastcore/foundation.py", line 105, in __init__
    items = listify(items, *rest, use_list=use_list, match=match)
  File "/Users/user1/.local/share/virtualenvs/get-twitter-likes-b1tRvag8/lib/python3.9/site-packages/fastcore/basics.py", line 56, in listify
    elif is_iter(o): res = list(o)
  File "/Users/user1/.local/share/virtualenvs/get-twitter-likes-b1tRvag8/lib/python3.9/site-packages/fastprogress/fastprogress.py", line 47, in __iter__
    raise e
  File "/Users/user1/.local/share/virtualenvs/get-twitter-likes-b1tRvag8/lib/python3.9/site-packages/fastprogress/fastprogress.py", line 41, in __iter__
    for i,o in enumerate(self.gen):
  File "/usr/local/Cellar/[email protected]/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
    yield fs.pop().result()
  File "/usr/local/Cellar/[email protected]/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/usr/local/Cellar/[email protected]/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/local/Cellar/[email protected]/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/user1/.local/share/virtualenvs/get-twitter-likes-b1tRvag8/lib/python3.9/site-packages/fastcore/parallel.py", line 54, in _call
    return g(item)
  File "/Users/user1/Projects/get-twitter-likes/src/core.py", line 183, in lookup_and_parse_tweets
    parsed_tweets = parse_tweets(statuses, output_format=output_format, parse_urls=parse_urls)
  File "/Users/user1/Projects/get-twitter-likes/src/core.py", line 190, in parse_tweets
    list_of_lists_of_parsed_tweets = parallel_map(parse_tweet,
  File "/Users/user1/Projects/get-twitter-likes/src/utils.py", line 20, in parallel_map
    return list(result)
  File "/usr/local/Cellar/[email protected]/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
    yield fs.pop().result()
  File "/usr/local/Cellar/[email protected]/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/Cellar/[email protected]/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/local/Cellar/[email protected]/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/user1/Projects/get-twitter-likes/src/core.py", line 85, in parse_tweet
    title_description = get_url_title_description(url, parse_url=parse_url)
  File "/Users/user1/Projects/get-twitter-likes/src/scraping.py", line 16, in get_url_title_description
    response = session.get(url, timeout=3)
  File "/Users/user1/.local/share/virtualenvs/get-twitter-likes-b1tRvag8/lib/python3.9/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/Users/user1/.local/share/virtualenvs/get-twitter-likes-b1tRvag8/lib/python3.9/site-packages/requests_cache/core.py", line 132, in request
    response = super(CachedSession, self).request(
  File "/Users/user1/.local/share/virtualenvs/get-twitter-likes-b1tRvag8/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/user1/.local/share/virtualenvs/get-twitter-likes-b1tRvag8/lib/python3.9/site-packages/requests_cache/core.py", line 109, in send
    return send_request_and_cache_response()
  File "/Users/user1/.local/share/virtualenvs/get-twitter-likes-b1tRvag8/lib/python3.9/site-packages/requests_cache/core.py", line 99, in send_request_and_cache_response
    self.cache.save_response(cache_key, response)
  File "/Users/user1/.local/share/virtualenvs/get-twitter-likes-b1tRvag8/lib/python3.9/site-packages/requests_cache/backends/base.py", line 46, in save_response
    self.responses[key] = self.reduce_response(response), datetime.utcnow()
  File "/Users/user1/.local/share/virtualenvs/get-twitter-likes-b1tRvag8/lib/python3.9/site-packages/requests_cache/backends/storage/dbdict.py", line 165, in __setitem__
    super(DbPickleDict, self).__setitem__(key,
  File "/Users/user1/.local/share/virtualenvs/get-twitter-likes-b1tRvag8/lib/python3.9/site-packages/requests_cache/backends/storage/dbdict.py", line 129, in __setitem__
    con.execute("insert or replace into `%s` (key,value) values (?,?)" %
sqlite3.OperationalError: database is locked

any ideas?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.