Expose underlying compression libraries directly about python-blosc HOT 6 CLOSED

I think what you are after is blosc_compress_ctx and blosc_decompress_ctx:

https://github.com/Blosc/c-blosc/blob/master/blosc/blosc.h#L184
https://github.com/Blosc/c-blosc/blob/master/blosc/blosc.h#L222

Using these with numinternalthreads=1 will allow you to use them from multithreaded apps without locks. Also, you can select different compressors like blosclz, lz4, lz4hc, snappy and zlib, and if you want to get rid of the internal blocking you can do that by using blocksize=nbytes.

Although these _ctx functions are not wrapped yet in python-blosc, they would be a nice addition and we would welcome a PR indeed.

from python-blosc.

Two things about the functions you are suggesting.

blosc.lzo.compress doesn't make sense right now, since we don't handle lzo.
blosc.shuffle I am not sure if this even makes sense with larger blocksizes

Also, there must be bindings for the compressors you want. Would it make sense to use those directly? Or are you looking for a uniform interface to them?

from python-blosc.

I'm looking for a uniform interface to the relevant options maintained by people that I trust care that it was done well.

from python-blosc.

Although these _ctx functions are not wrapped yet in python-blosc, they would be a nice addition and we would welcome a PR indeed.

From the comment here, it sounds like init can't be called if you want to use the context functions? If this is the case, then some workaround would be needed to avoid calling init in blosc/__init__.py on import.

Edit: From reading the source, it appears that there's no harm in using the *_ctx calls after blosc_init has been called, except that a global threadpool is created and never used.

from python-blosc.

@jcrist that sounds right.

from python-blosc.

@mrocklin I am assuming it is O.K. to close this since Dask now has compression support.

from python-blosc.