json.JSONDecoder causes Environment.begin to throw ReadersFullError about py-lmdb HOT 2 OPEN

This is an object lifetime problem that is exposing a subtle pylmdb bug. Keeping a reference to txn in BadDecoder causes a delayed finalization of txn.

As a temporary workaround, I found that adding decoder.txn = None after decoder = BadDecoder(txn) fixes it.

The performant solution is to create the txn context outside the loop.

I will investigate further.

from py-lmdb.

I've encountered ReadersFullError in a different context from the one I posted above. The error occurs if I read from and write to a single Environment using more than one process. I didn't even try to isolate the error, but I did manage a workaround, which is basically "turn it off and on again".

If Environment.begin() throws ReadersFullError, then I do a "soft reset" in the process where the error occurred: I call Environment.close() followed immediately by a call to lmdb.open(). In my test cases, doing soft resets worked if I ran on only two processes. (A soft reset wasn't needed at all for a single process.)

A soft reset could fail in one of two ways: Either lmdb.open() throws ReadersFullError or the first call to Environment.begin() does. If a soft reset fails in any process, I do a "hard reset":

1. Every (alive) process calls Environment.close(),
2. Every process waits at a multiprocessing.Event,
3. Once every alive process is waiting, a single process deletes the lockfile (lock.mdb),
4. The same process calls lmdb.open (which recreates the lockfile),
5. The remaining processes call lmdb.open.

from py-lmdb.