Comments (7)
Hi @aparamon, we've just merged a set of changes to fix building TileDB-Py against the current TileDB dev branch. It looks like the test files you sent are no longer available, so I couldn't test your example myself. Please let us know if you have a chance to test again.
from tiledb-py.
We are looking into this, thanks for the detailed bug report!
from tiledb-py.
Hi @stavrospapadopoulos, did you have a chance to look into this?
from tiledb-py.
Apologies for the delay. This is next in my list. I'll get back to you very soon.
from tiledb-py.
@aparamon I've just merged a fix for this. Thanks for reporting it and apologies for the delay. When you get the chance, please build/install TileDB (dev
) and TileDB-Py from source and let us know if the issue is fixed. We will add it in the next point release (hopefully in the next few days).
from tiledb-py.
Hi @stavrospapadopoulos, thank you for working on this!
Unfortunately, the version from PyPI still fails for me.
Installing from source didn't work either:
tiledb/libtiledb.cpp: In function ‘PyObject* __pyx_pf_6tiledb_9libtiledb_22consolidate(PyObject*, __pyx_obj_6tiledb_9libtiledb_Ctx*, PyObject*, PyObject*)’:
tiledb/libtiledb.cpp:63518:140: error: too few arguments to function ‘int32_t tiledb_array_consolidate_with_key(tiledb_ctx_t*, const char*, tiledb_encryption_type_t, const void*, uint32_t, tiledb_config_t*)’
__pyx_v_rc = tiledb_array_consolidate_with_key(__pyx_v_ctx_ptr, __pyx_v_uri_ptr, __pyx_v_key_type, __pyx_v_key_ptr, __pyx_v_key_len);
^
In file included from tiledb/libtiledb.cpp:628:0:
/c/data/tiledb/TileDB-Py/build/TileDB-dev/dist/include/tiledb/tiledb.h:3185:23: note: declared here
TILEDB_EXPORT int32_t tiledb_array_consolidate_with_key(
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
from tiledb-py.
I tested with a modified version of the original code, generating random data, and cannot reproduce a segfault with TileDB 1.6.0 / TileDB-Py 0.4.3.
Closing for now, but will certainly re-open if needed. Thanks for the issue report.
code:
from: #81
#%%
import sys, os
import tiledb
import numpy as np
ctx = tiledb.Ctx()
#if '--init' in sys.argv:
if True:
if os.path.isdir('spec'):
tiledb.remove('spec')
filters = tiledb.FilterList([tiledb.LZ4Filter()])
dom = tiledb.Domain(tiledb.Dim(name='scan', domain=(1, 506), tile=20, dtype=float),
tiledb.Dim(name='mz', domain=(0, 2000), tile=10, dtype=float))
schema = tiledb.ArraySchema(domain=dom, sparse=True, capacity=1024,
attrs=[tiledb.Attr(name='intensity', dtype=float, filters=filters)])
spec = tiledb.SparseArray.create('spec', schema)
with tiledb.SparseArray('spec', mode='w') as spec:
#npoints = 27_464_448
#scan_arr = np.zeros(npoints)
#mz_arr = np.zeros(npoints)
#intens_arr = np.zeros(npoints)
tiledb.stats_enable()
n_scans = 1_000_000
scan_arr = np.repeat(np.arange(1,507), 1000)
mz_arr = np.tile(np.arange(0,1000,step=1), 506)
intens_arr = np.random.rand(1000*506)
#i = 0
#scan = 0
#for line in open('nefazodone.raw'):
# line = line.strip()
# if line.startswith('RetTime='):
# scan += 1
# elif line.startswith('Mz='):
# mz, intens = line.split('=')[1].split(' ')
# scan_arr[i] = scan
# mz_arr[i] = float(mz)
# intens_arr[i] = float(intens)
# i += 1
spec[scan_arr, mz_arr] = {'intensity': intens_arr}
#assert i == npoints
tiledb.stats_dump()
tiledb.stats_disable()
#%%
with tiledb.SparseArray('spec', mode='r') as spec:
print(spec.nonempty_domain())
print(spec.domain)
tiledb.stats_enable()
data = spec[9.5:10.5, :]
tiledb.stats_dump()
tiledb.stats_disable()
print(data['intensity'])
tiledb.stats_enable()
data = spec[468.5:469.5, :]
tiledb.stats_dump()
tiledb.stats_disable()
print(data['intensity'])
#%%
</details>
from tiledb-py.
Related Issues (20)
- Nightly GitHub Actions Build Fail on Fri, June 16th 2023
- Nightly GitHub Actions Build Fail on Sun, June 18th 2023
- Nightly GitHub Actions Build Fail on Tue, June 20th 2023
- Nightly GitHub Actions Build Fail on Wed, June 21st 2023
- Nightly GitHub Actions Build Fail on Thu, June 22nd 2023
- Nightly GitHub Actions Build Fail on Fri, June 23rd 2023
- Nightly GitHub Actions Build Fail on Sat, June 24th 2023
- Nightly GitHub Actions Build Fail on Sun, June 25th 2023
- Nightly GitHub Actions Build Fail on Mon, June 26th 2023
- ValueError: shape mismatch HOT 3
- Nightly GitHub Actions Build Fail on Thu, August 17th 2023 HOT 1
- Feature: Support `pathlib.Path` HOT 2
- Nightly GitHub Actions Build Fail HOT 21
- Unable to access array from S3
- The nightly build job failed on Sunday (2023-09-17) HOT 124
- tile parameter ignored in variable length dimensions HOT 1
- tiledb-py > 0.21.6 and corresponding versions of tiledb embedded miss the path separator and query string prefix when utilizing SAS tokens with Azure Blob Storage HOT 1
- tiledb.walk() hangs indefinitely when blob with folder name exists HOT 1
- Error when reading from HDFS backend HOT 3
- Show enumerated value-types in enum-printer
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tiledb-py.