dogsheep / pocket-to-sqlite Goto Github PK

View Code? Open in Web Editor NEW

100.0 8.0 5.0 25 KB

Create a SQLite database containing data from your Pocket account

License: Apache License 2.0

Python 100.00%

sqlite pocket-api pocket dogsheep datasette datasette-io datasette-tool

pocket-to-sqlite's Introduction

Dogsheep

Dogsheep is a collection of tools for personal analytics using SQLite and Datasette.

Big internet companies know a lot about us. By exporting that data back out of them we can see what they know and maybe learn something interesting about ourselves.

Read more about Dogsheep on my blog: simonwillison.net/tags/dogsheep

Watch Personal Data Warehouses: Reclaiming Your Data for a demo of Dogsheep in action.

Dogsheep tools

These tools, maintained by the Dogsheep project, let you export your data into a SQLite database for further analysis.

twitter-to-sqlite uses the Twitter API or Twitter's "export your data" zip archive to create a database of your tweets, followers, people you follow, favourited tweets and much more besides.
healthkit-to-sqlite converts your Apple HealthKit export into a database, with details of your heart rate, workouts, step count and more.
swarm-to-sqlite imports your checkin history from Foursquare Swarm.
apple-notes-to-sqlite saves Apple Notes data to a database.
inaturalist-to-sqlite imports the wildlife and plants you have spotted on iNaturalist.
google-takeout-to-sqlite imports data from Google Takeout, which can include your Google Maps location history, your search history on Google and more to come.
genome-to-sqlite imports your genome from 23andMe into a SQLite database.
github-to-sqlite imports repositories you have created or starred on GitHub.
pocket-to-sqlite imports articles you have saved using Pocket.
hacker-news-to-sqlite imports stories and comments posted to Hacker News.
dogsheep-photos imports metadata about photos from Apple Photos, including machine learning labels. See Using SQL to find my best photo of a pelican according to Apple Photos.
dogsheep-beta is a search index which combines data from all of the other sources into a single faceted search interface.
evernote-to-sqlite converts exported Evernote files in the ENEX format into a SQLite database.

Tools by other developers

These tools help bring the Dogsheep philosophy to life.

beeminder-to-sqlite by Ben Congdon saves your goals and data points from Beeminder.
goodreads-to-sqlite by Tobias Kunze imports your reading history from Goodreads.
pinboard-to-sqlite by Jacob Kaplan-Moss saves your bookmarks from Pinboard.
todoist-to-sqlite by Ben Congdon saves your tasks and projects from Todoist.
parkrun-to-sqlite by Mark Woodbridge imports your parkruns.

pocket-to-sqlite's People

Contributors

Stargazers

Watchers

Forkers

jmnickerson05 tlockney inos-soft bgrins empjustine

pocket-to-sqlite's Issues

Store authentication information as "pocket_access_token" etc

The pocket_ prefix will mean that the same auth.json file can be used for other Dogsheep tools without Pocket over-riding a value set by some other tool.

Track and use the 'since' value

Pocket says:

Whenever possible, you should use the since parameter, or count and and offset parameters when retrieving a user's list. After retrieving the list, you should store the current time (which is provided along with the list response) and pass that in the next request for the list. This way the server only needs to return a small set (changes since that time) instead of the user's entire list every time.

At the bottom of https://getpocket.com/developer/docs/v3/retrieve

ISO timestamps

The time_added, time_updated and time_read columns currently store data like this:

September 19, 2019 - 00:30:30 UTC

Should use ISO instead, e.g. 2020-07-26T01:05:24+00:00

This may be solved or a feature already, but I couldn't figure it out, is it possible to extract and store also full text from the saved pages? The same way that Pocket parses the text, it'd be amazing to be able to store (and thus make searchable later) the text.

Thank you very much for the project, it's such an amazing idea!

-a option is used for "--auth" and for "--all"

I'm not sure which option is best, instead of -a -all.

Extract images into separate tables

As already done with authors. Slightly harder because images do not have a universally unique ID. Also need to figure out what to do about there being columns for both image and images.

Switch to pyproject.toml

First of my CLI tools to use https://til.simonwillison.net/python/pyproject

Set up a demo

First off, thanks for open sourcing this application! This is a suggestion to increase the amount of people that would make use of it: an example in the readme file would help.

Currently, users have to clone the app, install it, authorize through pocket, run a command, an then find out if this application does what they hope it does.

Another possibility is to add a file example-output.db, containing one (mock) Pocket article.

Keep up the good work!

When running `auth` command, don't overwrite an existing auth.json file

Ran the auth command in the same directory I'd previously set up an auth.json file for twitter-to-sqlite and it was completely overwritten. Not the biggest issue, but still unexpected. Ideally, for me, the keys would just be added to the existing file, but getting a warning and a chance to back out would be a good solution as well.

Use better pagination (and implement progress bar)

Right now we attempt to load everything at once - which caps out at 5,000 items and is really slow.

We can do better by implementing pagination using count and offset.

SSL Error

Here's the error I get when running pip install pocket-to-sqlite:

  Could not fetch URL https://pypi.python.org/simple/pocket-to-sqlite/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661) - skipping
  Could not find a version that satisfies the requirement pocket-to-sqlite (from versions: )
No matching distribution found for pocket-to-sqlite

Does this require python 3?