ghing / congressional-tweets Goto Github PK

It's my understanding that the timeline endpoints give you a subset of tweets while the streaming API lets you capture all (or more?) of the tweets.

Besides, don't we want to be informed in real time?

To do:

• Better handling of killing the app, probably using signal
• Use systemd to supervise and optionally restart the tweet streamer and loader
• Write command to stream tweets
• Write command to display streamed tweets
• Write command to load streamed tweets
• Deploy everything

Running the search_tweets command takes a long time:

time congressional_tweets search_tweets ".*Town Hall.*"
real    1m30.557s
user    0m2.980s
sys     0m0.168s

Even when specifying the --since-id option, it's pretty slow:

time congressional_tweets search_tweets --since-id 869255179597697024  ".*Town Hall.*" 
real    0m34.698s
user    0m0.388s
sys     0m0.060s

Proposed solution

At this point, I think using Text Search will offer better performance than using an index.

Tasks

• Create a text index. Migrations help make this explicit, so do this using migrations
• Update search_tweets command to use a $search query instead of a regex.

The streaming API returns tweets that mention the screen names we specify, which isn't bad, but it makes it harder to search through the tweets. search_tweets already has a --screen-name argument, but we need to allow specifying multiple screen names.

Once a significant number of tweets start appearing in the database, the performance of search_tweets degrades since it returns all matching tweets in the database. Add a way to filter on date, or if that doesn't work, on tweet ID since they're incremental to speed up searches and limit the number of results.

Rather than reading screen names from standard input, make them positional arguments. This makes it easier to specify the screen names from an environment variable.

Searching the tweets of Appalachian reps shows the latest Tweet as being May 25. I first noticed this on Friday or Saturday of last week (May 26 or May 27).

I've tried restarting the services:

sudo service stream_tweets restart
sudo service load_streamed_tweets restart

I also confirmed that the stream_tweets service was running by running the display_streamed_tweets command and got output.