Code Monkey home page Code Monkey logo

Comments (7)

ihnorton avatar ihnorton commented on June 6, 2024 1

Hi @aparamon, we've just merged a set of changes to fix building TileDB-Py against the current TileDB dev branch. It looks like the test files you sent are no longer available, so I couldn't test your example myself. Please let us know if you have a chance to test again.

from tiledb-py.

stavrospapadopoulos avatar stavrospapadopoulos commented on June 6, 2024

We are looking into this, thanks for the detailed bug report!

from tiledb-py.

aparamon avatar aparamon commented on June 6, 2024

Hi @stavrospapadopoulos, did you have a chance to look into this?

from tiledb-py.

stavrospapadopoulos avatar stavrospapadopoulos commented on June 6, 2024

Apologies for the delay. This is next in my list. I'll get back to you very soon.

from tiledb-py.

stavrospapadopoulos avatar stavrospapadopoulos commented on June 6, 2024

@aparamon I've just merged a fix for this. Thanks for reporting it and apologies for the delay. When you get the chance, please build/install TileDB (dev) and TileDB-Py from source and let us know if the issue is fixed. We will add it in the next point release (hopefully in the next few days).

from tiledb-py.

aparamon avatar aparamon commented on June 6, 2024

Hi @stavrospapadopoulos, thank you for working on this!

Unfortunately, the version from PyPI still fails for me.
Installing from source didn't work either:

tiledb/libtiledb.cpp: In function ‘PyObject* __pyx_pf_6tiledb_9libtiledb_22consolidate(PyObject*, __pyx_obj_6tiledb_9libtiledb_Ctx*, PyObject*, PyObject*)’:
tiledb/libtiledb.cpp:63518:140: error: too few arguments to function ‘int32_t tiledb_array_consolidate_with_key(tiledb_ctx_t*, const char*, tiledb_encryption_type_t, const void*, uint32_t, tiledb_config_t*)’
         __pyx_v_rc = tiledb_array_consolidate_with_key(__pyx_v_ctx_ptr, __pyx_v_uri_ptr, __pyx_v_key_type, __pyx_v_key_ptr, __pyx_v_key_len);
                                                                                                                                            ^
In file included from tiledb/libtiledb.cpp:628:0:
/c/data/tiledb/TileDB-Py/build/TileDB-dev/dist/include/tiledb/tiledb.h:3185:23: note: declared here
 TILEDB_EXPORT int32_t tiledb_array_consolidate_with_key(
                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

from tiledb-py.

ihnorton avatar ihnorton commented on June 6, 2024

I tested with a modified version of the original code, generating random data, and cannot reproduce a segfault with TileDB 1.6.0 / TileDB-Py 0.4.3.

Closing for now, but will certainly re-open if needed. Thanks for the issue report.

code:

```

from: #81

#%%
import sys, os
import tiledb
import numpy as np

ctx = tiledb.Ctx()

#if '--init' in sys.argv:
if True:
if os.path.isdir('spec'):
tiledb.remove('spec')

filters = tiledb.FilterList([tiledb.LZ4Filter()])
dom = tiledb.Domain(tiledb.Dim(name='scan', domain=(1, 506), tile=20, dtype=float),
                    tiledb.Dim(name='mz', domain=(0, 2000), tile=10, dtype=float))
schema = tiledb.ArraySchema(domain=dom, sparse=True, capacity=1024,
                            attrs=[tiledb.Attr(name='intensity', dtype=float, filters=filters)])
spec = tiledb.SparseArray.create('spec', schema)

with tiledb.SparseArray('spec', mode='w') as spec:
    #npoints = 27_464_448
    #scan_arr = np.zeros(npoints)
    #mz_arr = np.zeros(npoints)
    #intens_arr = np.zeros(npoints)

    tiledb.stats_enable()

    n_scans = 1_000_000
    scan_arr = np.repeat(np.arange(1,507), 1000)
    mz_arr = np.tile(np.arange(0,1000,step=1), 506)
    intens_arr = np.random.rand(1000*506)

    #i = 0
    #scan = 0
    #for line in open('nefazodone.raw'):
    #    line = line.strip()
    #    if line.startswith('RetTime='):
    #        scan += 1
    #    elif line.startswith('Mz='):
    #        mz, intens = line.split('=')[1].split(' ')
    #        scan_arr[i] = scan
    #        mz_arr[i] = float(mz)
    #        intens_arr[i] = float(intens)
    #        i += 1

    spec[scan_arr, mz_arr] = {'intensity': intens_arr}
    #assert i == npoints
    tiledb.stats_dump()
    tiledb.stats_disable()

#%%
with tiledb.SparseArray('spec', mode='r') as spec:
print(spec.nonempty_domain())
print(spec.domain)
tiledb.stats_enable()
data = spec[9.5:10.5, :]
tiledb.stats_dump()
tiledb.stats_disable()
print(data['intensity'])

tiledb.stats_enable()
data = spec[468.5:469.5, :]
tiledb.stats_dump()
tiledb.stats_disable()
print(data['intensity'])

#%%


</details>

from tiledb-py.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.