coleifer / sophy Goto Github PK

View Code? Open in Web Editor NEW

80.0 80.0 7.0 690 KB

Fast Python bindings to Sophia Database

Home Page: http://sophy.readthedocs.io/en/latest/

License: Other

Python 3.81% C 92.21% Cython 3.99%

cython embedded-database nosql python sophia

sophy's People

Contributors

Stargazers

Watchers

Forkers

muravjov romannekhor smithsamuelm catroot mosquito ra2003

sophy's Issues

Project License

The sophia lisence is BSD. But I didn't find out any license mention for the sophy.

Not sure if there is a special method in the API (i didn't find one) but i do have some issues (segfaults to be accurate) when i try to close sophia instance through close method defined here https://github.com/coleifer/sophy/blob/master/sophy.pyx#L545

This issue came out of a problem with Flask's code reloader which opened too many files resulting in
Exception Exception: Exception("sophia/log/sl.c:61 log file '' open error: Too many open files",) in 'sophy.Sophia.open' ignored

How to re-open existing database?

Creating the database works fine using the example given at https://sophy.readthedocs.io/en/latest/api.html#Database , and it's stored to disk. And i can read values to and from it, at that moment.

But, when using get_database only, commenting out the add_database line, wanting to just reopen the stored existing database afterwards without overwriting it with a new (and then empty) one, i get this:

kv_db = env.get_database('kv') File "sophy.pyx", line 251, in sophy.Sophia.get_database KeyError: 'kv'

Is there something i'm doing incorrectly or have misunderstood?

My 'env' folder looks like this:

test/kv
test/kv/00000000000000000001.db
test/kv/scheme

Any help or tips are appreciated.

Segfault

Just installed sophy using the git repo. Created a simple database and exited resulting in a segfault. Ran again and database appeared OK, but segfaulted again on exit. OS X El Capitan.

ipython
Python 2.7.10 (default, Oct 23 2015, 18:05:06)
Type "copyright", "credits" or "license" for more information.

IPython 4.0.3 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: from sophy import Sophia

In [2]: env = Sophia('/tmp/sophia-env', [('test-db', 'string')])

In [3]: db = env['test-db']

In [4]: env.open()
Out[4]: False

In [5]: db = env['test-db']

In [6]: db['k3']
Out[6]: '5'

In [7]: list(db[::True])
Out[7]: [('k3', '5'), ('k2', 'v2'), ('k1', 'v1')]

In [8]: env.close()
Out[8]: True

In [9]:
Do you really want to exit ([y]/n)?
Segmentation fault: 11

all the value i get from sophy is None while key was correct

here is my test.py content

#!/usr/bin/env python
# coding: utf-8

from sophy import *
from random import randint, choice

CS = 'abcdefghijklmnopqrstuvwxyz'
genstr = lambda n=4: ''.join(choice(CS) for idx in xrange(n))


def random_insert():
        env  = Sophia('./data/cache')
        schema = Schema(key_parts=[StringIndex('key')], value_parts=[StringIndex('val')])
        db = env.add_database('poi', schema)


        env.open()
        for idx in xrange(10):
                k, v = genstr(6), str(randint(0, 9999))
                db[k] = v
        env.close()

def show():
        env  = Sophia('./data/cache')
        schema = Schema(key_parts=[StringIndex('key')], value_parts=[StringIndex('val')])
        db = env.add_database('poi', schema)


        env.open()
        for k,v in db:
                print k,v

        env.close()

show()

and this is the output of that show()

ubuntu@devops:/data_ext/jboard$ python ./test.py 
a None
cksbtp None
hixdik None
iogfbz None
mufzys None
rjqnyt None
rrujgz None
sxbwno None
ueulqi None
xbfeff None
yzouqe None

Querying a part of index and value?

I wonder if there is possibility to query a part of complex indexes or even query for value?
Suppose we have a schema:

schema = Schema(key_parts=[StringIndex('key'), StringIndex('predicate')],
value_parts=[StringIndex('value')])

Since we have defined StringIndex for value and for key and predicate, can i query for a part of index or for specific value?

Unable to create new database. KeyError: 'db-test'

sophy version is current git HEAD. Python 3.4.3

This error occurs when I try to create a new database or run the test suite.

sophy git:(master) ✗ python tests.py
Exception ignored in: 'sophy._BaseDBObject.open'
MemoryError: Unable to allocate object: <sophy.Database object at 0x7fb40aa92348>.
EException ignored in: 'sophy._BaseDBObject.open'
MemoryError: Unable to allocate object: <sophy.Database object at 0x7fb40aaacd68>.

........(lots of these errors)

ERROR: test_view (__main__.TestU32Index)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 40, in setUp
    super(BaseSophiaTestMethods, self).setUp()
  File "tests.py", line 21, in setUp
    self.db = self.sophia[DB_NAME]
  File "sophy.pyx", line 240, in sophy.Sophia.__getitem__ (sophy.c:6545)
    return self.dbs[name]
KeyError: 'db-test'

----------------------------------------------------------------------
Ran 39 tests in 0.076s

FAILED (errors=39)

How to limit the memory used by sophia

Héllo,

I'd like to use sophia for my sotoki project to replace sqlite, pgsql or wiredtiger.

Here is my code:

            from sophy import Sophia
            from msgpack import loads, dumps

            env = Sophia(db, auto_open=False)
            env.memory_limit = 1024 * 1024 * 1024
            env.open()

            posts = env.create_database('Post', 'u32')


            filepath = os.path.join(dump, klass.filename())
            for data in iterate('dump'):
                 posts[int(data['Id'])] = dumps(data)
            for key, data in posts.cursor():
                uid = key
                values = loads(data)
            env.close()

But sophia seem to use all the memory available even if I've set a memory limit. What I am doing wrong?

writing, closing, and then reopening a Sophia db in one Python session results in segfault on exit

If I create and close a Sophia db and then try to open and read from it in the same Python session, Python segfaults on exit (i.e., after printing the db contents in the example below). Creating the db in one session and then opening and reading from it in another succeeds without any segfault. I'm using bddf3da with Python 2.7.12 on MacOS 10.11.6:

import sophy

# assumes that /tmp/foo does not exist before the script is run
env = sophy.Sophia('/tmp/foo', [('foo', 'string')])
db = env['foo']
db['a'] = 'xxx'
db['b'] = 'yyy'
db['c'] = 'zzz'
env.close()

env = sophy.Sophia('/tmp/foo', [('foo', 'string')])
db = env['foo']
for (k, v) in db:
    print k, v
env.close()

Segmentatoin fault when key is U32Index

Hello. I trying to create database with U32Index as a key.

from sophy import Sophia, Schema, BytesIndex, U32Index

env = Sophia('/tmp')

schema = Schema(
    key_parts=[U32Index('key')],
    value_parts=[BytesIndex('value')]
)

db = env.add_database('test_num_key', schema)

if not env.open():
    raise RuntimeError('Unable to open Sophia environment.')

but interpreter fails.

BTW: cool binding, great work!

Can't create new environment

When I run the following code:

import sophy
# assumes that /tmp/foo does not exist before the script is run
env = Sophia('/tmp/foo')
db = env.create_database('mydb', 'string')

I get the following error:

Exception ignored in: 'sophy.Sophia.open'
Traceback (most recent call last):
  File "sophy.pyx", line 154, in sophy._ConfigManager.apply_all (sophy.c:6382)
  File "sophy.pyx", line 150, in sophy._ConfigManager.apply_config (sophy.c:6215)
  File "sophy.pyx", line 73, in sophy._check (sophy.c:3704)
Exception: sophia/runtime/sr_conf.c:339 bad configuration path: db.b'foo'.mmap

Is there any configuration I should do?

Import Error undefined symbol: PyString_FromStringAndSize

Looks like there is a python 3 issue.

Using Python 3.4.3

In [1]: import sophy
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-75851a3a749b> in <module>()
----> 1 import sophy

ImportError: /home/vagrant/miniconda/envs/env/lib/python3.4/site-packages/sophy.cpython-34m.so: undefined symbol: PyString_FromStringAndSize

Install logs:

Collecting Cython
Collecting sophy
  Downloading sophy-0.1.2.tar.gz (297kB)
    100% |████████████████████████████████| 299kB 765kB/s
Building wheels for collected packages: sophy
  Running setup.py bdist_wheel for sophy
  Stored in directory: /home/vagrant/.cache/pip/wheels/ed/72/4d/36acd1b75237dd93ebf51be25bea18d011dd72ad71fe6a8751
Successfully built sophy
Installing collected packages: Cython, sophy
Successfully installed Cython-0.23.4 sophy-0.1.2

Encoding problems?

Hi,

I used virtualenv to set up a Python 3.4.3; then did "pip install cython", cloned master branch of sophy (commit 2cac2f5 as HEAD), followed by successful "python setup.py build" and "python setup.py install".

When running "python tests.py", all tests fail due to:
Exception: sophia/runtime/sr_conf.c:339 bad configuration path: db.b'db-test'.mmap

Full output (all test cases fail in the same way):

ERROR: test_version (main.TestConfiguration)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 20, in setUp
    self.sophia = self.create_env()
  File "tests.py", line 29, in create_env
    return Sophia(TEST_DIR, [(DB_NAME, self._index_type)])
  File "sophy.pyx", line 208, in sophy.Sophia.init (sophy.c:6464)
    self.open()
  File "sophy.pyx", line 226, in sophy.Sophia.open (sophy.c:6733)
    self.config.apply_all()
  File "sophy.pyx", line 159, in sophy._ConfigManager.apply_all (sophy.c:5877)
    self.apply_config(key, value)
  File "sophy.pyx", line 155, in sophy._ConfigManager.apply_config (sophy.c:5707)
    _check(self.sophia.handle, rc)
  File "sophy.pyx", line 78, in sophy._check (sophy.c:3297)
    raise Exception(error)
Exception: sophia/runtime/sr_conf.c:339 bad configuration path: db.b'db-test'.mmap

Same b'db-test' appears when building from the tag 0.1.6.

Not sure what causes this. Looks like something with encoding/decoding... maybe whatever is encoded must be decoded when used later for string concatenation (and subsequent re-encode).

I also wonder if this is somehow an issue with locales. Mine are set to sv_SE.UTF-8.

Python3 ?

Any estimate of when Python3 support will be happening?

unbound memory problem

Origin: https://groups.google.com/forum/#!topic/sophia-database/nX5HbdF7mcE

Using sophy bindings and sophia 2.1 I try to load a several Gb file into a database but even after settings the memory limit the consumption grows without limit.

What I am doing wrong?

Here's more or less the code I use:

            from sophy import Sophia
            from msgpack import loads, dumps

            env = Sophia(db, auto_open=False)
            env.memory_limit = 1024 * 1024 * 1024
            env.open()

            posts = env.create_database('Post', 'u32')

            filepath = os.path.join('dump', 'Posts.xml')
            for data in iterate(filepath):
                 posts[int(data['Id'])] = dumps(data)
            env.close()

Can we please check that memory.limit is set correctly?
Thanks!