This is likely a pickle/cPickle bug in Python 2.7 when using HIGHEST_PROTOCOL which is diskcache default. Found while working on nexB/scancode-toolkit#267
You can see these two tests in action:

• bare pickling issue: nexB/scancode-toolkit@50bfc21
• same issue when using diskcache: nexB/scancode-toolkit@6b92ccf

The simple workaround is to avoid using HIGHEST_PROTOCOL. I guess this can be done by passing a custom Disk subclass forced to use no specific protocol.

File "/home/grantj/repos/xxx/env27/lib/python2.7/site-packages/diskcache/djangocache.py", line 77, in delete
    self._cache.delete(key)
  File "/home/grantj/repos/xxx/env27/lib/python2.7/site-packages/diskcache/fanout.py", line 123, in delete
    return self.__delitem__(key)
  File "/home/grantj/repos/xxx/env27/lib/python2.7/site-packages/diskcache/fanout.py", line 111, in __delitem__
    return self._shards[index].__delitem__(key)
  File "/home/grantj/repos/xxx/env27/lib/python2.7/site-packages/diskcache/core.py", line 737, in __delitem__
    return self._delete(rowid, version, filename)
  File "/home/grantj/repos/xxx/env27/lib/python2.7/site-packages/diskcache/core.py", line 763, in _delete
    self._remove(filename)
  File "/home/grantj/repos/xxx/env27/lib/python2.7/site-packages/diskcache/core.py", line 790, in _remove
    os.remove(full_path)

OSError: [Errno 13] Permission denied: '/tmp/xxx-cache/003/e6/be/87b78c58d79c1a502b05ce645db8.val'

This is an odd error. On one particular server I get transient [Errno 13] Permission denied errors.

Occurred today in Travis at 1.6.3 on both CPython 3.4 and PyPy.

======================================================================
ERROR: Stress test multiple threads and processes.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/pypy-2.5.0/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/travis/build/grantjenks/python-diskcache/tests/stress_test_fanout.py", line 280, in stress_test_mp
    stress_test()
  File "/home/travis/build/grantjenks/python-diskcache/tests/stress_test_fanout.py", line 244, in stress_test
    cache.check()
  File "/home/travis/build/grantjenks/python-diskcache/diskcache/fanout.py", line 135, in check
    return sum((shard.check(fix=fix) for shard in self._shards), [])
  File "/home/travis/build/grantjenks/python-diskcache/diskcache/fanout.py", line 135, in <genexpr>
    return sum((shard.check(fix=fix) for shard in self._shards), [])
  File "/home/travis/build/grantjenks/python-diskcache/diskcache/core.py", line 873, in check
    warnings.warn('file not found: %s' % full_path)
UserWarning: file not found: tmp/000/90/06/d93b748a5ebea1fee3fc3fe441bf.val

This is worrisome. I don't yet understand how a file was lost.

Add atomic increment and decrement methods.

Upcoming features:

• Issue #51 fixed
• Issue #52 add cull() method
• Issue #54 need to optimize pickle
• Issue #58 need tests for dir-copy backup
• Pull request #61
• Issue #59 need support for new-style .format strings when culling

'evict' method should not create an index. Instead:

• Create the index during initialization.
• Provide create_tag_index method.
• Provide drop_tag_index method.

First of all thanks for releasing this! Super cool library, I was just reading about your benchmarks and wondering why culling (glob) was needed (and why it's slow)? The reason you "cull" is to check if you need to remove some old keys to make space for some new set(), is that right?

Would it be possible to keep track of old file state somehow instead of having to keep globbing? Or making many smaller folders to make the sets cheaper... not sure I understand how all this works, and I'm sure you've thought of this I'm just curious why it's not the way-to-go. Thanks for any explanation, I hope to use this on some fun projects soon :)

I get this error but this kinda random. Only found on python 2.7.3 on Linux/Ubuntu 12.04.

You can see the code calling it here:
https://github.com/nexB/scancode-toolkit/blob/0030a717f8f202b24aa6943f4d62d66a627fd047/src/licensedcode/cache.py#L148

[...]
 File "/home/pombreda/scancode-toolkit/src/licensedcode/cache.py", line 171, in get
   cached = self.cache.get(cache_key)
 File "/home/pombreda/scancode-toolkit/local/lib/python2.7/site-packages/diskcache/fanout.py", line 84, in get
   default=default, read=read, expire_time=expire_time, tag=tag,
 File "/home/pombreda/scancode-toolkit/local/lib/python2.7/site-packages/diskcache/core.py", line 664, in get
   value = self._disk.fetch(self._dir, mode, filename, db_value, read)
 File "/home/pombreda/scancode-toolkit/local/lib/python2.7/site-packages/diskcache/core.py", line 245, in fetch
   return pickle.load(io.BytesIO(value))
TypeError: 'buffer' does not have the buffer interface

I am using 1.6.6
https://github.com/nexB/scancode-toolkit/blob/0030a717f8f202b24aa6943f4d62d66a627fd047/thirdparty/prod/diskcache.ABOUT

Add "cas" or "compare and set" method. This is useful for making atomic things.

Use consistent hashing for string, bytes, and datetime objects. Python 3 initializes PYTHONHASHSEED to a random value. That really messes with things. Use a different hash technique.

As described in DiskCache Development it would be great to have an lru_cache-like decorator to cache function calls.

I contacted Yuval Greenfield, the author of filecache and he said the missing decorator feature was what blocked him from using the module. His decorator accepts one argument, seconds before expiry.

I prototyped something simple in Issue #8. It should be relatively easy to combine it with the source from Python 3's functools.lru_cache

In [119]: cache.add(1,True)
Out[119]: True

In [120]: cache.add(1,True)
Out[120]: False

In [121]: cache.add(1,False)
Out[121]: False

In [122]: cache.add(1,False)
Out[122]: False

This API choice reminds me of C. Why not return -1 for the sake of the good old times? An exception would have been a much better choice.

Include atomic "add" method, like "set" but abort if already exists.

Is there a builtin way to list keys in the cache?

Right now I'm using

cache._sql('SELECT key FROM Cache').fetchall()

Hey,

would you accept a pull request that adds a max_age parameter to .get(). So that it returns a default value if the cached value is too old.

I guess something like this is not possible yet, or?

Define __iter__ and __reversed__.

A couple things I've noticed in programming:

1. If I store bytes in the cache, it may choose to store the result in the database (if it's small enough). When I read those bytes back, I may request read=True and depending on the size of the value, I may get back a bytes object or a file handle. It seems preferable to always receive a file-like object. This is not possible for integer and floating point types but I think that's ok.
2. If I try to retrieve a value with read=True then often I would prefer to raise KeyError than return the default. So instead of using this pattern:

try:
    with cache.get('key', read=True) as reader:
        filename = reader.name
except AttributeError: # `None` does not implement the context manager protocol.
    filename = None

I would prefer this pattern:

try:
    with cache.get('key', read=True) as reader:
        filename = reader.name
except KeyError: # Clearly the key was not found in the cache.
    filename = None

But this changes the semantics of get when read=True. Should read=True always return a file-like object? If it used StringIO or BytesIO to do so, then the code above would break on small values because there is no filename (unless it was explicitly stored using read=True).

I'm starting to regret the read=True extension to get and set methods. Maybe the names read and write would have been better. Something like:

cache.write('key', file_like_object)

And:

cache.read('key') # Return file handle

I'm interested in feedback from others. If you use this functionality, please contribute your opinion.

Pseudocode:

def rate_limit(key, count, seconds, memo={}):
    """Rate limit resource identified by key to count per seconds.

    >>> while True:
    ...     wait = next(rate_limit('some key', 5, 8))
    ...     if not wait:
    ...         break
    ...     else:
    ...         time.sleep(wait)
    ... do_rate_limited_work()

    """
    if (key, count, seconds) in memo:
        return memo[key, count, seconds]

    assert isinstance(count, int) and count > 0

    def meter():
        rate = float(count) / seconds
        last = time.time()
        tally = count

        while True:
            now = time.time()
            extent = now - last
            last = now
            tally += extent * rate

            if tally > count:
                tally = count
            elif tally < 1:
                yield (1 - tally) / rate
            else:
                tally -= 1
                yield 0

    limiter = meter()

    memo[key, count, seconds] = limiter
    return limiter

Need to track last and tally in Cache. Might be better to build as a separate object. The generator pattern is not that useful. But the algorithm is tested and correct.

FanoutCache Subdirectories Created with Mode 700. Prefer 755.

It would be nice if incremental backup behavior were guaranteed by tests to always work.

Edit: I would like to have there be tests that check that using rsync to back up a cache dir will keep the backup up-to-date with all new content in the cache, and that only new content is synched (so that if 99% of the content is unchanged, it will only be scanned locally by rsync, and not redundantly copied).

If the backed-up copy of the cache dir is then used, it should behave identically to the original.

For our use cases we can pause all readers and writers during the course of the backup, so we're not concerned about race conditions.

... for instance with Appveyor. Windows multi-processing and locking is a tad different .... and this is an understatement.

Document File-cache Features:

• set(..., read=True, ...)
• read(...)

Close must accept kwargs because some projects bind close as signal handler

for example dbtemplates, sorl-thumbnail etc..

signals.request_finished.connect(cache.close)

fixed in #6

Hi,
I'm currently using joblib for caching numpy array objects. Is there any benchmark on these kind of inputs for DiskCache?

Hi

Is there a way to set the default expire time to be used in .set methods for diskcache.Cache objects?

Thanks

Reminder: Add Tutorial docs for diskcache.Deque and diskcache.Index.

Document incompatibility of DiskCache and NFS due to SQLite.

This is relevant to PythonAnywhere users.

This morning in Travis:

======================================================================
ERROR: Stress test multiple threads and processes.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/travis/build/grantjenks/python-diskcache/tests/stress_test_fanout.py", line 280, in stress_test_mp
    stress_test()
  File "/home/travis/build/grantjenks/python-diskcache/tests/stress_test_fanout.py", line 244, in stress_test
    cache.check()
  File "/home/travis/build/grantjenks/python-diskcache/diskcache/fanout.py", line 135, in check
    return sum((shard.check(fix=fix) for shard in self._shards), [])
  File "/home/travis/build/grantjenks/python-diskcache/diskcache/fanout.py", line 135, in <genexpr>
    return sum((shard.check(fix=fix) for shard in self._shards), [])
  File "/home/travis/build/grantjenks/python-diskcache/diskcache/core.py", line 841, in check
    rowid, self._disk.get(key, raw)
UserWarning: row 10 partially commited with key (0.8579395219751248, 0.8579395219751248, 0.8579395219751248, 0.8579395219751248, 0.8579395219751248, 0.8579395219751248, 0.8579395219751248, 0.8579395219751248, 0.8579395219751248, 0.8579395219751248, 0.8579395219751248)
----------------------------------------------------------------------

I'm not sure how that happened. What would cause two threads/processes to abort the same set operation in DiskCache? Maybe SQLite would do so?

FanoutCache leverages the connection timeout. And the "set" and "add" methods do eviction after their write. So the write may succeed and then the eviction may timeout. If the eviction times-out then FanoutCache will report False for an unsuccessful operation although the write occurred successfully.

This odd behavior is worth documenting.

API pseudocode:

def push(channel, value, expire=None, read=False, tag=None):
    # return stored key
    ...
def pull(channel, default=None, expire_time=False, tag=False):
    # does not support "read" because (key, value) is deleted
    # return value
    ...

• Queue support is problematic because its slow. DiskCache does best when the ratio of reads to writes is 10:1 or greater. Queues are closer to 1:1 without backpressure.
• TODO: Support channel length?
• TODO: Support channel iteration?

Change DiskCache Index.setdefault() to use loop { get(); add() }. Current implementation is not atomic.

Which leads to errors like this:

➜  sv-tools  nix-shell                                                                         ~/dev/sv-tools
these derivations will be built:
  /nix/store/ilvmx3q5bfjqmhc1sarzrlwvbvfc4rly-python-diskcache-v1.6.6-src.drv
  /nix/store/8lmqbi0j8xc2xhvgvl3xqldixgibsg5x-python3.5-diskcache-1.6.6.drv
building path(s) ‘/nix/store/999v4wbl7ycvg3355p66sfnjjj01k2rs-python-diskcache-v1.6.6-src’

trying https://github.com/grantjenks/python-diskcache/archive/v1.6.6.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   135    0   135    0     0    258      0 --:--:-- --:--:-- --:--:--   258
100  406k  100  406k    0     0   262k      0  0:00:01  0:00:01 --:--:-- 1278k
unpacking source archive /tmp/nix-build-python-diskcache-v1.6.6-src.drv-0/v1.6.6.tar.gz
building path(s) ‘/nix/store/cxsbysaqka3312a5wlmpwwdxn8l1zsc4-python3.5-diskcache-1.6.6’
unpacking sources
unpacking source archive /nix/store/999v4wbl7ycvg3355p66sfnjjj01k2rs-python-diskcache-v1.6.6-src
source root is python-diskcache-v1.6.6-src
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file python-diskcache-v1.6.6-src/tox.ini
patching sources
configuring
building
Traceback (most recent call last):
  File "nix_run_setup.py", line 6, in <module>
    exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
  File "setup.py", line 19, in <module>
    readme = reader.read()
  File "/nix/store/w9gwivwr7ih57b4y1dfhv64xy316p5qn-python3-3.5.1/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 1068: ordinal not in range(128)
builder for ‘/nix/store/8lmqbi0j8xc2xhvgvl3xqldixgibsg5x-python3.5-diskcache-1.6.6.drv’ failed with exit code 1
error: build of ‘/nix/store/8lmqbi0j8xc2xhvgvl3xqldixgibsg5x-python3.5-diskcache-1.6.6.drv’ failed
/run/current-system/sw/bin/nix-shell: failed to build all dependencies

A quick fix would be to set the encoding to utf-8 on open.

Pickling is tricky because a file handle is maintained to the database. But ultimately that's just a filename and a sqlite3 connect call. Seems possible that dunder getstate and setstate could be used to make Cache objects pickle-able.

Increase large value threshold based on https://www.sqlite.org/intern-v-extern-blob.html

Also worth setting the page_size explicitly. Locally the default was 4K but in production the default was 1K. I think 4K sounds good enough.

I would also guess that our use case will prefer larger blobs because reading a file requires first reading the filename from the database. It's probably worth creating a benchmark for testing.

• Add eviction policy: 'error' - raise exception if cache volume exceeds size limit. No 'init' and no 'get' values.

Using diskcache 2.9.0, via python 2.7 on Mac:

I am having problems using a tuple of strings as a cache key. Once the key has been placed into the cache and is subsequently retrieved from the cache via the key iterator, it apparently no longer matches the pickled form of the key when it was first inserted, leading to the value not being found when looked up in sqlite. I have tried changing the pickle format version to no avail. Using a list instead of a tuple works properly. Here is code to demonstrate the problem:

from diskcache import Cache, Index
import shutil
import tempfile

key_part_0 = u"part0"
key_part_1 = u"part1"
to_test = [
    (key_part_0, key_part_1),
    [key_part_0, key_part_1]
]
for key in to_test:
    tmpdir = tempfile.mkdtemp(prefix="discache_test.")
    try:
        dc = Index.fromcache(Cache(tmpdir, eviction_policy='none', size_limit=2**32)) #4GB
        dc[key] = {
            "example0": ["value0"]
        }
        diskcache_key = dc.keys()[0]
        print "Keys equal: %s" % (diskcache_key==key)
        print "Value using original key: %s" % dc[key]
        try:
            print "Value using retrieved key: %s" % dc[diskcache_key]
        except KeyError as e:
            print "Could not find value using retrieved key"
    finally:
        shutil.rmtree(tmpdir, True)

And the output:

Keys equal: True
Value using original key: {'example0': ['value0']}
Could not find value using retrieved key
Keys equal: True
Value using original key: {'example0': ['value0']}
Value using retrieved key: {'example0': ['value0']}

Thanks for any assistance you can provide.

Helps a lot with lots of small values. My use case:

• batch queries to API
• caching calls is inefficient since sets of queries vary
• caching by each query separately makes sense

So it looks like:

def _mygene_fetch(queries, specie):
    # Read cache
    cache = diskcache.Cache('/tmp/mygene-cache/' + specie, timeout=CACHE_TIMEOUT)
    res = compact({q: cache.get(q) for q in queries})

    queries = set(queries) - set(res)
    if queries:
        data = api.querymany(queries, species=specie)
        new = {item['query']: (item['entrezgene'], item['symbol'])
               for item in data}
        res.update(new)
        # Cache results
        for k, v in new.iteritems():
            cache[k] = v

    return res

And it's slow.

I lost the stack trace but once on Travis-CI, I saw an OperationalError: database schema has changed bug. It was triggered during stress at "CREATE TABLE IF NOT EXISTS Settings ..."

Some research indicated that perhaps a separate cache process had changed the underlying schema and interacted badly with a statement cache. I'm not sure how that works.

This Issue is open as a reminder. If anyone sees something similar, would you add notes here?

Grant,
I couldn't find a way to contact you regarding a question about DiskCache, except to file an issue.
I do not know how to cull items from the cache if the cache size is larger than the size_limit.

If I fill the cache to something larger than the size_limit, the cache does not report any items as expired, so calling the expire method does nothing. How is the size_limit parameter supposed to work?

Though DiskCache has a dictionary-like interface, the hash protocol is not used. Neither the __hash__ nor __eq__ methods are used for lookups. Instead it depends on the serialization method. For strings, bytes, integers, and floats it works the same. But large integers and other Python objects will be pickled. The bytes representation of pickling will be used for equality. Hashing is not used at all.

We created a simple transaction class in memory which worked great but it consumes too much heap memory so we switched to diskcache 2.0.2 with Python 3.5. In most cases, diskcache works as expected. However, on occasion we get a pickling exception. Is this a bug? Is there a workaround? Here is the trace, notice the first relationship successfully pushes to the cache, the second one fails:

2016-09-24 12:03:42,371 INFO chemGraph: relationship(108494) Article-id-chem-article-10651284-[chemical]-Chemical-id-chem-chemical-thromboplastin
2016-09-24 12:03:42,461 INFO chemGraph: relationship(108495) Article-id-chem-article-10651284-[mesh]-MESH-id-umls-d008958
ERROR:root:transaction cache failed: (f0e044f)-[:mesh {id:"chem-article-10651284-umls-d008958",importDate:"2016-09-24T16:03:42.461744Z",nameHash:"chem-article-10651284-umls-d008958",source:"http://www.chemicals-r-us.info/chem",type:"article-mesh",uuid:"6de965a7-1f40-5d65-ae5a-ee7ab9723bbf"}]->(a011ffb)
Traceback (most recent call last):
  File "/chem/Fusion/ChemGraph.py", line 240, in checkTransactionLimits
    self.transaction_cache[name]=node
  File "/usr/lib/python3.5/site-packages/diskcache/core.py", line 510, in set
    size, mode, filename, db_value = self._disk.store(value, read)
  File "/usr/lib/python3.5/site-packages/diskcache/core.py", line 196, in store
    result = pickle.dumps(value, protocol=self._protocol)
_pickle.PicklingError: args[0] from __newobj__ args has the wrong class

Here is our diskcache initialization:

  from diskcache import Cache
  self.transaction_cache=Cache(cachePath+'/transaction/'+index, size_limit=MaxCacheSize)

Iteration Skips MAX(rowid)

Hi, i'm building a small tool with Diskcache to calculate/store/compare image hashes. My current implementation uses a FanoutCache to memoize different hashes under different parameters and the intention is to use the hashes for distance calculation.
The result would be sets of image-paths with (closely) similar hashes (image deduplication).

I'm now wondering what the correct approach would be to get results back from the cache to run my similarity algorithm. As far as i understand the docs i could retrieve an index from the current fanout cache and iterate all values, but directly working with the SQLlite disk might be more interesting because i would be able to express my algorithms in SQL directly.

Do you have any thoughts or possible preferred approach you could enlighten me with?

Thank you in advance!

I had to do an emergency delete of the disk cache /var/tmp/django_disk_cache as my server had run out of disk space.

Ever since I receive Django errors

Exception Type: ValueError at x Exception Value: Key ':1xxx' not found

Disabling DjangoCache is the ony current fix. I've checked the Django db and there's no corresponding table for DjangoCache. There's obviously some reference to these keys somewhere but I can't find them. The docs make mention to a sqlite db, but I've searched my installation and can't find it.

The help for DjangoCache also mention a clear command, which I assume I'm meant to run in a python shell, but I can't figure out how to run it and there's no examples.

The current Cache __iter__ only iterates on keys and therefore iterating the whole cache requires getting each item one by one. I want a dict-like iteritems() method that efficiently iterates and yields (key, value) instead.

For some applications, I would like to generate cache offline (ie during week end, for week day application).
and copy the cache folder to another server/PC where the cache can be used.

Is there any limitation or is it platform dependant ?
(ex Windows ---> Linux) .

What are the limit of size for diskcarch ?
Nb of Key, Size of each individual file (caching 1Go pandas df is ok ?)

Edit: After further consideration, I realized that what I'm really looking for is more akin to a LRU eviction policy that only evicts if the last access_time is more than N (in our case, 90 days).

How difficult is it to add custom expiration policies?

Add persistent mapping.

Use http://locust.io/ to create a more "realistic" Django benchmark.

For reference, see email from [email protected].

import diskcache
cache = diskcache.Cache("tmp")
@cache.memoize
def fn(x):
    return x

AttributeError: 'Cache' object has no attribute 'memoize'

Is there a reason it is only implemented for FanoutCache?