Hi from python-isal about deflate HOT 4 CLOSED

One possible improvement I see: when decompressing I see here a void * is created which is later copied into a bytes object with PyBytes_FromStringAndSize.
But you can also do

PyObject * return_value = PyBytes_FromStringAndSize(NULL, decompressed_size);
void * decompressed_data = (void *)PyBytes_AS_STRING(return_value); 
// Decompression set up here
libdeflate_gzip_decompress(
        decompressor, data.buf, data.len, decompressed_data, size, &decompressed_size);
// error-handling code here
return return_value;

This way you only allocate a output buffer once for the bytes object. No copying required.

from deflate.

Thanks for the suggestion! It wasn't quite that simple, since decompressed_size isn't known before decompression, but there is a _PyBytes_Resize function. Almost certainly better than a copy.

I wrote this to use in https://github.com/imsweb/pzip, which compresses (and encrypts) in chunks, so I had no need for a streaming interface -- libdeflate is very well suited for this case.

from deflate.

t wasn't quite that simple, since decompressed_size isn't known before decompression

Well it should be equal to the ISIZE block from the gzip trailer. Otherwise the gzip is corrupt. So you already initiate the buffer with the correct size. And the nice thing is that _PyBytes_Resize quits early when the size is already correct. So no resizing happens in the correct case.

I wrote this to use in https://github.com/imsweb/pzip, which compresses (and encrypts) in chunks, so I had no need for a streaming interface -- libdeflate is very well suited for this case.

Ah very useful. Chunked compression is also used by a format in bioinformatics called BAM. It uses block gzip format, which is basically compressed blocks. The length of the compressed block is saved in the first EXTRA field, while the length of the decompressed result is saved in ISIZE. This is very useful as you know the exact sizes of the buffers.

from deflate.

I just got a notification (I comaintain the conda-feedstock for libdeflate) https://github.com/ebiggers/libdeflate/releases/tag/v1.9. FYI.

from deflate.