Code Monkey home page Code Monkey logo

sqlitedict's Introduction

sqlitedict -- persistent dict, backed-up by SQLite and pickle

Travis License

A lightweight wrapper around Python's sqlite3 database with a simple, Pythonic dict-like interface and support for multi-thread access:

>>> from sqlitedict import SqliteDict
>>> mydict = SqliteDict('./my_db.sqlite', autocommit=True)
>>> mydict['some_key'] = 'any_picklable_object'
>>> print(mydict['some_key'])  # prints the new value
any_picklable_object
>>> for key, value in mydict.iteritems():
...     print(key, value)
some_key any_picklable_object
>>> print(len(mydict)) # etc... all dict functions work
1
>>> mydict.close()

Pickle is used internally to (de)serialize the values. Keys are arbitrary strings, values arbitrary pickle-able objects.

If you don't use autocommit (default is no autocommit for performance), then don't forget to call mydict.commit() when done with a transaction:

>>> # using SqliteDict as context manager works too (RECOMMENDED)
>>> with SqliteDict('./my_db.sqlite') as mydict:  # note no autocommit=True
...     mydict['some_key'] = u"first value"
...     mydict['another_key'] = range(10)
...     mydict.commit()
...     mydict['some_key'] = u"new value"
...     # no explicit commit here
>>> with SqliteDict('./my_db.sqlite') as mydict:  # re-open the same DB
...     print(mydict['some_key'])  # outputs 'first value', not 'new value'
first value

Features

  • Values can be any picklable objects (uses cPickle with the highest protocol).

  • Support for multiple tables (=dicts) living in the same database file.

  • Support for access from multiple threads to the same connection (needed by e.g. Pyro). Vanilla sqlite3 gives you ProgrammingError: SQLite objects created in a thread can only be used in that same thread.

    Concurrent requests are still serialized internally, so this "multithreaded support" doesn't give you any performance benefits. It is a work-around for sqlite limitations in Python.

  • Support for custom serialization or compression:

# use JSON instead of pickle
>>> import json
>>> mydict = SqliteDict('./my_db.sqlite', encode=json.dumps, decode=json.loads)

# apply zlib compression after pickling
>>> import zlib, pickle, sqlite3
>>> def my_encode(obj):
...     return sqlite3.Binary(zlib.compress(pickle.dumps(obj, pickle.HIGHEST_PROTOCOL)))
>>> def my_decode(obj):
...     return pickle.loads(zlib.decompress(bytes(obj)))
>>> mydict = SqliteDict('./my_db.sqlite', encode=my_encode, decode=my_decode)

Installation

The module has no dependencies beyond Python itself. The minimum Python version is 2.5, continuously tested on Python 2.7, and above on on Github Actions.

Install or upgrade with:

pip install -U sqlitedict

or from the source tar.gz:

python setup.py install

Documentation

Standard Python document strings are inside the module:

import sqlitedict
help(sqlitedict)

(but it's just dict with a commit, really).

Beware: because of Python semantics, sqlitedict cannot know when a mutable SqliteDict-backed entry was modified in RAM. For example, mydict.setdefault('new_key', []).append(1) will leave mydict['new_key'] equal to empty list, not [1]. You'll need to explicitly assign the mutated object back to SqliteDict to achieve the same effect:

>>> val = mydict.get('new_key', [])
>>> val.append(1)  # sqlite DB not updated here!
>>> mydict['new_key'] = val  # now updated

For developers

Install:

# pip install pytest coverage pytest-coverage

To perform all tests:

# mkdir -p tests/db
# pytest tests

To perform all tests with coverage:

# pytest tests --cov=sqlitedict

Comments, bug reports

sqlitedict resides on github. You can file issues or pull requests there.

Housekeeping

>>> import os
>>> os.unlink('my_db.sqlite')

sqlitedict is open source software released under the Apache 2.0 license. Copyright (c) 2011-now Radim Řehůřek and contributors.

sqlitedict's People

Contributors

and34 avatar bit avatar da4089 avatar digenis avatar ecederstrand avatar endlisnis avatar ergoithz avatar hholst80 avatar jquast avatar jtatum avatar menshikh-iv avatar mpenkov avatar nrhinehart avatar ownport avatar piskvorky avatar pombredanne avatar shoyu63 avatar thechief389 avatar tirkarthi avatar tmylk avatar toddrme2178 avatar transfluxus avatar vcalv avatar vrescobar avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.