IntelDeflater intermittently fails to properly compress outputs with GKL 0.8.8 about gkl HOT 3 CLOSED

ISA-L added a new configuration field (isal_zstream->hist_bits) between versions 2.21 and 2.30 which wasn't initialized to default value by isal_deflate_stateless_init call. This was fixed by a recent commit in ISA-L (intel/isa-l@9f2b68f).

When there were multiple simultaneous allocations of memory using malloc the OS kernel could provide a previously used memory page which contained a value different than default in place corresponding to hist_bits field. Values of hist_bits between 1 and 14 (especially lower ones) could reduce compression efficiency for this particular compression session. Other values (including 0) are replaced by ISA-l with 15 which is the default, most efficient setting.

Using calloc instead of malloc to allocate and fill with zeroes the isal_zstream struct also fixes this issue and prevents similar issues happening in the future.

from gkl.

Thank you for reaching out. I noticed between GKL 0.8.6 to 0.8.8, ISAL has been upgraded from 2.21 to 2.30 but since it is happening intermittently we will first try to reproduce it on our end using a small bam file(~9gb) like mentioned in the original issue. We will use a 7gb file from our old long reads list (PAE09121_dae79b.bam.raw) and iterate it for ~100 times to check any anomalies in output sizes.

from gkl.

@mateuszsnowak Yay!

from gkl.