jamesls / semidbm Goto Github PK

Using Python 3.7's shelve with the default dbm I run into the same size limitation noted here http://jamesls.com/semidbm-a-pure-python-dbm.html (notably HASH: Out of overflow pages. Increase page size) using a Mac. Having installed gdbm it won't appear with my Conda Pythons.

semidbm came to the rescue using the following code snippet. The class and function are lifted directly from Python's shelve.py. I see no speed difference but I do see a an ability to scale to more objects that dbm lacked. gdbm should have provided a similar solution but on my Anaconda distribution I can't get it to work ( for reference import dbm.gnu generates ModuleNotFoundError: No module named '_gdbm').

Thank you for this package! I hope that the snippet below helps other who use shelve on a large dataset.

from shelve import Shelf
class DbfilenameShelfSemidbm(Shelf):
    """Shelf implementation using the "dbm" generic dbm interface.

    This is initialized with the filename for the dbm database.
    See the module's __doc__ string for an overview of the interface.
    """

    def __init__(self, filename, flag='c', protocol=None, writeback=False):
        import dbm
        Shelf.__init__(self, semidbm.open(filename, flag), protocol, writeback)


def open_semidbm(filename, flag='c', protocol=None, writeback=False):
    """Open a persistent dictionary for reading and writing.

    The filename parameter is the base filename for the underlying
    database.  As a side-effect, an extension may be added to the
    filename and more than one file may be created.  The optional flag
    parameter has the same interpretation as the flag parameter of
    dbm.open(). The optional protocol parameter specifies the
    version of the pickle protocol.

    See the module's __doc__ string for an overview of the interface.
    """

    return DbfilenameShelfSemidbm(filename, flag, protocol, writeback)

Timing on a smaller client task (prior to the HASH error above):

• dbm inside a default shelve 1m20
• semidbm inside the derived shelve 1m20 (the same as dbm)

In google cloud function,
If you call semidbm.db.open( filename, "r" ), it happens that error 「Function failed on loading user code. Error message: [Errno 30] Read-only file system: '/env/local/lib/python3.7/site-packages/pykakasi/kanwadict3.db/data'」
( It happens in pykakasi.)
The cause of that is 「compat.DATA_OPEN_FLAGS」.
Please modify.

I think I'd be fine with supporting just python3.3 for now.

I need to convert dbhash because it's not supported in python3, so in python2 I:

• try to open as semidb
• if fail, open as dbhash, copy keys to new-name, close, rename.

This lets the conversion happen automagically in my db code, rather than needing explicit conversion. BUT the rename fails because the file has not been closed, so in mmapload.py I made the change:

        try:    # Phil added try...except
            header = f.read(8)
            self._verify_header(header)
        except:
            f.close()
            raise

I did not try to submit a change because it may be better wrapped in the other try clause.

Hi @jamesls and team,

Looks like there has not been a release for near 5 years now. Are there any plans to address this, specially merging open PRs to fix bugs, fixing the CI failures, etc?

If not, mind giving access to new contributors (github, travis, pypi.org) to revive the project and unblock dependent projects?

(cc @oxij, @sylvinus, @speedplane, @AnythingTechPro, @miurahr)

The latest semidbm version from your git repo does not build successfully:

$ python setup.py build
running build
running build_py
file semidbm.py (for module semidbm) not found
file semidbm.py (for module semidbm) not found

Installation also reports this error during sudo python setup.py install. The resulting semidbm cannot be imported.

Is there a release of 0.4 available that does build successfully? Or another git repo where it builds? Or is there an easy fix to this build problem?

Not officially released yet, but semidbm should support jython2.7. This gives jython users an easy to use dbm for jython (gdbm is not available on jython).

In terms of the work, we'll have to refactor the use of mmap, as that't not available on the jython platform.

It should error out or return the correct result.

(Pdb) self.store["blah"] = "1"
(Pdb) "blah" in self.store
False
(Pdb) self.store["blah"]
b'1'
(Pdb) self.store
<semidbm.db._SemiDBM object at 0x11f6ec050>
(Pdb) semidbm.__version__
'0.5.1'

>>> import semidbm
>>> db = semidbm.open('test', 'n')
>>> del db[b'nonexistent']
Traceback [...]
KeyError: b'nonexistent'
>>> db[b'foo'] = b'bar'
>>> db[b'foo']
b'exi'
>>> db.close()
>>> db = semidbm.open('test', 'r')
Traceback [...]
KeyError: b'nonexistent'

The fix is probably to move the del above the write in _SemiDBM.__delitem__ (and add a regression test).

Hello,
Congrats for this nice project.

pip install semidbm

does not work on Python 3.7.

Is it planned to update it on PyPI?

Thanks!

I have a dbm I use that I access via read-only sdbm handle in a web server, I never write to this handle (which wouldn't be possible anyway), but every once in awhile my data disappears, i.e. all the contents disappear (size goes from 2.6G to 8kb). It's hard to debug this because I don't write to the handle anywhere and it happens once every few weeks so it's hard for me to reproduce. I do access the read_only handle from multiple threads, so I'm wondering if somehow that's causing a problem? Here is how I create the handle

sdbm_read_only = sdbm.open(os.path.expandvars(self.config.get("sdbm_location")),'r')

Consider switching to a binary format for the db, it make it more efficient to load and possibly improve performance.

Confirmed this only happens in windows py26. Seems to be an issue with remapping pages once we exceed the number of mapped pages.

======================================================================
ERROR: test_remap_required (test_semidbm.TestRemapping)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Administrator\Documents\GitHub\semidbm\test_semidbm.py", line 330, in test_remap_required
    db2 = self.open_db_file()
  File "C:\Users\Administrator\Documents\GitHub\semidbm\test_semidbm.py", line 26, in open_db_file
    return semidbm.open(self.dbdir, 'c', **kwargs)
  File "C:\Users\Administrator\Documents\GitHub\semidbm\semidbm\db.py", line 461, in open
    return _SemiDBM(filename, **kwargs)
  File "C:\Users\Administrator\Documents\GitHub\semidbm\semidbm\db.py", line 73, in __init__
    self._load_db()
  File "C:\Users\Administrator\Documents\GitHub\semidbm\semidbm\db.py", line 81, in _load_db
    self._index = self._load_index(self._data_filename)
  File "C:\Users\Administrator\Documents\GitHub\semidbm\semidbm\db.py", line 92, in _load_index
    return self._load_index_from_fileobj(filename)
  File "C:\Users\Administrator\Documents\GitHub\semidbm\semidbm\db.py", line 105, in _load_index_from_fileobj
    for key_name, offset, size in self._read_index(filename):
  File "C:\Users\Administrator\Documents\GitHub\semidbm\semidbm\db.py", line 163, in _read_index
    offset=num_resizes * remap_size)
WindowsError: [Error 8] Not enough storage is available to process this command

----------------------------------------------------------------------
Ran 93 tests in 0.952s

FAILED (errors=1)

This will make it easier to determine whether or not we can load the db file.

notably .items()?