For broadinstitute/gatk#2302

See: https://mvnrepository.com/artifact/com.intel.gkl/gkl. Latest is 0.8.8. Also, the release name has a v in it. Thanks for the quick patch updating log4j!

We need to free the memory allocated for huffman tables on this line of IntelDeflater.cc, unless the ISA-L code is doing that for us.

For such a low-level library, I'd rather not have it as a dependency (so it will get rolled into my assembly jar), and certainly not have its logging info on the console. Can we just get rid of it and log to a file or somewhere else?

If yes, is there any introduction, or any benchmark?

We suspect there may be performance and/or correctness bugs when running the GKL with long reads data, so we'd like to do some systematic testing of the library (particularly the deflater/inflater code) to determine what the state of long reads support is.

@jonn and/or someone on the long reads team at the Broad can provide suitable test data

@kvg reports that running the Intel deflater via GATK on long reads data intermittently produces corrupt bam outputs. His specific use case is sharding a single unaligned bam file into multiple smaller bams. Running with the JDK deflater (--use-jdk-deflater) appears to resolve the issue.

Example error when trying to read a corrupt shard (reading with htsjdk produces the same error):

$ java -jar gatk.jar SplitSubreadsByZmw -I sharding_test.bam -O intel_compression/v1/ -nr 100000
$ java -jar gatk.jar SplitSubreadsByZmw -I sharding_test.bam -O intel_compression/v2/ -nr 100000
$ java -jar gatk.jar SplitSubreadsByZmw -I sharding_test.bam -O intel_compression/v3/ -nr 100000
$ java -jar gatk.jar SplitSubreadsByZmw -I sharding_test.bam -O intel_compression/v4/ -nr 100000
$ java -jar gatk.jar SplitSubreadsByZmw -I sharding_test.bam -O intel_compression/v5/ -nr 100000

$ samtools view intel_compression/v1/sharding_test.000002.bam > /dev/null
$ samtools view intel_compression/v2/sharding_test.000002.bam > /dev/null
[E::bgzf_read] Read block operation failed with error 2 after 30675 of 72043 bytes
[main_samview] truncated file.
$ samtools view intel_compression/v3/sharding_test.000002.bam > /dev/null
$ samtools view intel_compression/v4/sharding_test.000002.bam > /dev/null
$ samtools view intel_compression/v5/sharding_test.000002.bam > /dev/null

(Only the second attempt yields a corrupted file; runs before and after appear to be correct, despite nothing changing between steps.)

There may be a bug in https://github.com/Intel-HLS/GKL/blob/master/src/main/native/compression/IntelDeflater.cc, perhaps triggered when a read spans many compressed blocks.

This bug is also tracked in the GATK repo: broadinstitute/gatk#5798

Add code to report:

• Number of OMP threads requested
• Number of OMP threads available
• Number of OMP threads used

If number of threads used is less than the number requested, generate a warning message.

Add Java routines to set and get the FTZ flag.

For broadinstitute/gatk#1771

We have received reports of GKL hanging while checking for AVX support.

It appears the grep -i avx /proc/cpuinfo process hangs forever in Java. However, the grep command runs fine under bash.

We should implement a more robust AVX check that does not rely on grep and works on multiple platforms.

For broadinstitute/gatk#1985

We need to check for AVX support in load and return false if AVX is not supported.

The IntelInflater API needs some changes to match the Inflater API.

1. The IntelInflater should not require calling reset() before use.
2. When inflate(result) is called without specifying the length argument, length should default to result.length.

The following code will test these changes.

        final IntelInflaterFactory intelInflaterFactory = new IntelInflaterFactory();

        for (int i = 0; i < 10; i++) {
            // create deflater with compression level i
            final Deflater deflater = new Deflater(i, true);

            // setup deflater
            deflater.reset();
            deflater.setInput(input);
            deflater.finish();

            // compress data
            int compressedBytes = 0;
            while (!deflater.finished()) {
                compressedBytes = deflater.deflate(compressed, 0, compressed.length);
            }
            deflater.end();

            // log results
            log.info("%d bytes compressed to %d bytes : %2.2f%% compression\n",
                    LEN, compressedBytes, 100.0 - 100.0 * compressedBytes / LEN);

            // decompress and check output == input
            Inflater inflater = intelInflaterFactory.makeInflater(true);
            try {
                inflater.setInput(compressed, 0, compressedBytes);
                inflater.inflate(result);
                inflater.end();
            } catch (java.util.zip.DataFormatException e) {
                e.printStackTrace();
            }

            Assert.assertEquals(input, result);
        }

I'm unable to load the GKL dylibs when running under Java 11. I'm not sure if this is specific to OSX or if it is true for linux too. I get the following error:

1:51:12.613 INFO  NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Users/louisb/Workspace/picard/build/install/picard/lib/gkl-0.8.5.jar!/com/intel/gkl/native/libgkl_compression.dylib
11:51:12.628 WARN  NativeLibraryLoader - Unable to load libgkl_compression.dylib from native/libgkl_compression.dylib (/private/var/folders/q3/hw5cxmn52wq347lg7rb_mzlw0000gq/T/louisb/libgkl_compression1725451455108434181.dylib: dlopen(/private/var/folders/q3/hw5cxmn52wq347lg7rb_mzlw0000gq/T/louisb/libgkl_compression1725451455108434181.dylib, 1): no suitable image found.  Did find:
	/private/var/folders/q3/hw5cxmn52wq347lg7rb_mzlw0000gq/T/louisb/libgkl_compression1725451455108434181.dylib: code signature in (/private/var/folders/q3/hw5cxmn52wq347lg7rb_mzlw0000gq/T/louisb/libgkl_compression1725451455108434181.dylib) not valid for use in process using Library Validation: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.)

It seems like this is a new "feature" from apple to prevent you from running any code that hasn't be submitted to apple for notarization, but it only seems to apply when running java 11 at the moment and not to java 8.

Enable multi-threaded AVX PairHMM using OpenMP, with a fallback to single-threaded mode when OpenMP is not supported.

For broadinstitute/gatk#1819

This is an expansion of the issue reported here: broadinstitute/gatk#7693.

In summary we observed that when running the standard GATK warp inputs on some of our gold standard test data (which allocates 8000MB machines and allocates ~6974MB of that to java -Xmx) we start getting no-error message machine failures when the job runs for ~6-7 hours. The failures all follow the pattern of abrupt terminations of java without any sort of error message and seem to occur on the same 6ish shards. Given that there is no typical java error (and the results of one of our tests) we suspect its a memory failure occurring outside of java which doesn't definitively point to the GKL but its suspect. We discovered this when we were experimenting with sharing the HaplotypeCaller to 10 shards rather than the default 50 that we typically run it with. That is to say that running the same sample that succeeds when sharded 50 ways (where each shard takes ~2 hours) over the same data nothing fails but when we increase the runtime and genomic space for GATK by 5x we get persistent and repeatable memory failures on over half the shards.

Some experiments we have tried to track this down and their results:
-Doubling the machine memory clears up the problem. This corresponds to allocating ~15GB to java xmx which it may or may not actually attempt to use over a Haplotype Caller run.
-Doubling the non-java memory also evidently clears up this crash (i.e. allocating ~5974MB to java -Xmx and leaving 2gb unallocated). This this more or less rules out java memory issues.
-We have tested across gatk 4.2.2.0 and 4.2.0.0 and the crashes affect both versions (for context we updated to GKL 0.8.8 in 4.2.1.0).

I am running with the standard WARP wdls on some standard data and I can share the exact tasks that are being run if need be. If there are any suggestions for techniques for tracking down if it really is the GKL (which we know often does take a significant amount of memory) or what else might be causing problems. We do expect the memory usage across sites to be spiky in gatk in general but its unexpected behavior that running over the same sites after running for 6 hours before should make a significant difference to the used memory.

For broadinstitute/gatk#1819

Hi Developer, when i install GKL, i meet an error. Could you help me solve the problem ?
Install command and error info as below:

~$./gradlew --stacktrace test
Caused by: java.io.EOFException: SSL peer shut down incorrectly

~$./gradlew --debug test
14:14:16.672 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Could not resolve all dependencies for configuration ':classpath'.
14:14:16.672 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Could not download org.eclipse.jgit.jar (org.eclipse.jgit:org.eclipse.jgit:4.1.1.201511131810-r)
14:14:16.673 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Could not get resource 'https://plugins.gradle.org/m2/org/eclipse/jgit/org.eclipse.jgit/4.1.1.201511131810-r/org.eclipse.jgit-4.1.1.201511131810-r.jar'.
14:14:16.673 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Could not GET 'https://plugins.gradle.org/m2/org/eclipse/jgit/org.eclipse.jgit/4.1.1.201511131810-r/org.eclipse.jgit-4.1.1.201511131810-r.jar'.
14:14:16.674 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Remote host closed connection during handshake

i find that i can download the org.eclipse.jgit-4.1.1.201511131810-r.jar from browser, but i don't know how to load the org.eclipse.jgit-4.1.1.201511131810-r.jar from local file system.

GATK has an intermittent failure that results in a core dump, which I think we've tracked down to native pair HMM. We're seeing it quite frequently on travis lately. AFAIK we have only ever seen this on Java 11 (not 8).

More details are available in broadinstitute/gatk#6649. Attached is a travis log that has the core dump log embedded.

log.txt

The recent log4j exploit has led me to checking various projects and realizing that many Java/JVM based projects include log4j solely due to having GKL at a dependency. I would argue that it is not good practise to have library code like the GKL bind directly to a logging framework like log4j, but instead to have it use commons-logging or something similar, that allows tools and applications that use the library to redirect logging information into the logging toolkit of their choice.

GKL is a fantastic library, but only available in X86-based architecture, Any plans to support ARM or PowerPC architecture optimization?

When compiling with gcc 11.2.0, smithWatermanBackTrack isn't actually inlined, and this results in both the AVX2 and AVX-512 implementations calling the same copy of it. This is bad because each copy needs to be specialized to use either AVX2 instructions or AVX-512 instructions. Here's the relevant bits from objdump -d build/native/libgkl_smithwaterman.so:

0000000000001610 <_Z19runSWOnePairBT_avx2iiiiPhS_ssaPciPjPi>:
    193c:       e8 8f f7 ff ff          call   10d0 <_Z22smithWatermanBackTrackP10dnaSeqPairiiiiPii@plt>

0000000000002a80 <_Z21runSWOnePairBT_avx512iiiiPhS_ssaPciPjPi>:
    2dec:       e8 df e2 ff ff          call   10d0 <_Z22smithWatermanBackTrackP10dnaSeqPairiiiiPii@plt>

I think it should at least be changed to static inline, and possibly renamed to CONCAT(smithWatermanBackTrack_,SIMD_ENGINE).

We have reports (https://gatk.broadinstitute.org/hc/en-us/community/posts/15590727717659-A-fatal-error-has-been-detected-by-the-Java-Runtime-Environment-when-running-Haplotypecaller-) that the GKL can crash on certain CPU architectures and/or with certain JVMs. Below is an excerpt of a core dump from a user running on an Intel(R) Xeon(R) Gold 6242R CPU with OpenJDK 64-Bit Server VM 17.0.3. Another user added the following:

"This particular error was quite prominent in all of my skylake/cascade lake class hardware especially when I used openjdk provided by the OS repository. Is it possible for you to try openjdk from eclipse temurin (www.adoptium.net). ? There are other options available as well like using the docker instance of gatk could help. Also you may want to try reducing the number of pairHMM threads to 1 instead of the default 4 or any other value being used by your command. AVX512 capable cpus from intel tend to have this issue with certain java builds."

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f799b0bda54, pid=268132, tid=268133
#
# JRE version: OpenJDK Runtime Environment (17.0.3) (build 17.0.3-internal+0-adhoc..src)
# Java VM: OpenJDK 64-Bit Server VM (17.0.3-internal+0-adhoc..src, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C  [libc.so.6+0x8fa54]  __memset_sse2+0x54
#
# Core dump will be written. Default location: /scale03/nikhil/augustus2/snpcalling1/93Y21/core.268132
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  S U M M A R Y ------------

Command Line: -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 /scale03/nikhil/envs/gatk4/share/gatk4-4.4.0.0-0/gatk-package-4.4.0.0-local.jar HaplotypeCaller -R Gmax_508_v4.0.fa -I 93Y21.bam -O rawvariants.vcf

Host: Intel(R) Xeon(R) Gold 6242R CPU @ 3.10GHz, 40 cores, 2636G, CentOS Linux release 7.9.2009 (Core)
Time: Thu May 18 10:14:49 2023 CDT elapsed time: 28701.945198 seconds (0d 7h 58m 21s)

---------------  T H R E A D  ---------------

Current thread (0x00007f79940146c0):  JavaThread "main" [_thread_in_native, id=268133, stack(0x00007f799b4b7000,0x00007f799b5b8000)]

Stack: [0x00007f799b4b7000,0x00007f799b5b8000],  sp=0x00007f799b462e28,  free space=18014398509481647k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.6+0x8fa54]  __memset_sse2+0x54
C  [libgkl_pairhmm_omp8599022423123915931.so+0x1500f]  Java_com_intel_gkl_pairhmm_IntelPairHmm_computeLikelihoodsNative._omp_fn.0+0xcf

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 7481  com.intel.gkl.pairhmm.IntelPairHmm.computeLikelihoodsNative([Ljava/lang/Object;[Ljava/lang/Object;[D)V (0 bytes) @ 0x00007f79858c7e9c [0x00007f79858c7e20+0x000000000000007c]
J 8402 c2 com.intel.gkl.pairhmm.IntelPairHmm.computeLikelihoods([Lorg/broadinstitute/gatk/nativebindings/pairhmm/ReadDataHolder;[Lorg/broadinstitute/gatk/nativebindings/pairhmm/HaplotypeDataHolder;[D)V (87 bytes) @ 0x00007f7985bbb200 [0x00007f7985bbb1c0+0x0000000000000040]
J 6576 c2 org.broadinstitute.hellbender.utils.pairhmm.VectorLoglessPairHMM.computeLog10Likelihoods(Lorg/broadinstitute/hellbender/utils/genotyper/LikelihoodMatrix;Ljava/util/List;Lorg/broadinstitute/hellbender/utils/pairhmm/PairHMMInputScoreImputator;)V (450 bytes) @ 0x00007f798575ef78 [0x00007f798575d840+0x0000000000001738]
J 7873 c2 org.broadinstitute.hellbender.tools.walkers.haplotypecaller.PairHMMLikelihoodCalculationEngine.computeReadLikelihoods(Lorg/broadinstitute/hellbender/utils/genotyper/LikelihoodMatrix;)V (132 bytes) @ 0x00007f79858c0fd4 [0x00007f79858bf100+0x0000000000001ed4]
J 8416 c2 org.broadinstitute.hellbender.tools.walkers.haplotypecaller.PairHMMLikelihoodCalculationEngine.computeReadLikelihoods(Lorg/broadinstitute/hellbender/tools/walkers/haplotypecaller/AssemblyResultSet;Lorg/broadinstitute/hellbender/utils/genotyper/SampleList;Ljava/util/Map;Z)Lorg/broadinstitute/hellbender/utils/genotyper/AlleleLikelihoods; (25 bytes) @ 0x00007f7985bc9850 [0x00007f7985bc70a0+0x00000000000027b0]
J 8280 c2 org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCallerEngine.callRegion(Lorg/broadinstitute/hellbender/engine/AssemblyRegion;Lorg/broadinstitute/hellbender/engine/FeatureContext;Lorg/broadinstitute/hellbender/engine/ReferenceContext;)Ljava/util/List; (1934 bytes) @ 0x00007f7985b3ca1c [0x00007f7985b3a120+0x00000000000028fc]
J 9114% c2 org.broadinstitute.hellbender.engine.AssemblyRegionWalker.processReadShard(Lorg/broadinstitute/hellbender/engine/MultiIntervalLocalReadShard;Lorg/broadinstitute/hellbender/engine/ReferenceDataSource;Lorg/broadinstitute/hellbender/engine/FeatureManager;)V (154 bytes) @ 0x00007f7985d843ac [0x00007f7985d83c00+0x00000000000007ac]
j  org.broadinstitute.hellbender.engine.AssemblyRegionWalker.traverse()V+83
j  org.broadinstitute.hellbender.engine.GATKTool.doWork()Ljava/lang/Object;+19
j  org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool()Ljava/lang/Object;+34
j  org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs()Ljava/lang/Object;+225
j  org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain([Ljava/lang/String;)Ljava/lang/Object;+14
j  org.broadinstitute.hellbender.Main.runCommandLineProgram(Lorg/broadinstitute/hellbender/cmdline/CommandLineProgram;[Ljava/lang/String;)Ljava/lang/Object;+20
j  org.broadinstitute.hellbender.Main.mainEntry([Ljava/lang/String;)V+22
j  org.broadinstitute.hellbender.Main.main([Ljava/lang/String;)V+8
v  ~StubRoutines::call_stub

siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 0x00007f799b4a97c0

Hi, I'm also missing some documentation how to make the system use these libraries... Do I need to set LD_LIBRARY_PATH? Do I have to move those files somewhere?

Originally posted by @TekTimmy in #93 (comment)

GKL's build embeds the 2.13.2 version of log4j 2 which is vulnerable to an exploit. Please update the build.gradle to use log4j 2 version 2.15.0 or newer.

I'm getting a segfault on a specific machine in the deflater code. See broadinstitute/picard#1383

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00000001203e0ea7, pid=31176, tid=0x0000000000003703
#
# JRE version: OpenJDK Runtime Environment (8.0_222-b10) (build 1.8.0_222-b10)
# Java VM: OpenJDK 64-Bit Server VM (25.222-b10 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# C  [libgkl_compression386528246230126242.dylib+0x6ea7]  deflate_medium+0x867
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/louisb/Workspace/picard/hs_err_pid31176.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Results: SUCCESS (0 tests, 0 successes, 0 failures, 0 skipped)

An example error log: hs_err_pid30548.log

It seems to be consistently reproducible on this specific laptop, but not yet on other machines.

See this code:

M_t_1_y should not be an exact copy of M_t_1, but instead it should be a copy prior to the vector shift. That means it should be initialized to VEC_SET1_VAL(zero) at the start of the stripe.

Background:
I have ported this code to Rust (https://github.com/philipc/gkl-rs), and I have been investigating why the values differ from the scalar implementation. In addition to this bug, there is also a difference due to use of approximations for the match-to-match probability. After addressing both of these, the Rust implementation produces values that very closely match the expected results in https://github.com/Intel-HLS/GKL/blob/master/src/test/resources/pairhmm-testdata.txt

It can be installed with 'yum install -y help2man' on Centos 7 OS.

No GATK-independent API is provided for the accelerated smith waterman implementation. It would be quite useful to be able to use this implementation to replace http://jaligner.sourceforge.net or the java bindings I wrote for https://github.com/mengyao/Complete-Striped-Smith-Waterman-Library.

Are there any plans to release a stand-alone AVX accelerated S/W implementation?

Hi Developer,
I meet some error when I install GKL. Could you help me to solve this problem?

Install command and error info as below:

root@cgvm-bk-r3:/pub/bin/GKL# ./gradlew test
:compileJava UP-TO-DATE
:cmakeConfig
:cmakeBuild
In file included from /pub/bin/GKL/src/main/native/pairhmm/avx-pairhmm.h:5:0,
from /pub/bin/GKL/src/main/native/pairhmm/avx_impl.cc:2:
/pub/bin/GKL/src/main/native/pairhmm/avx-types.h:8:17: error: ‘__m512’ does not name a type
ALIGNED __m512 ALIGNED d;
^
/pub/bin/GKL/src/main/native/pairhmm/avx-types.h:11:17: error: ‘__m512i’ does not name a type
ALIGNED __m512i ALIGNED i;
^
/pub/bin/GKL/src/main/native/pairhmm/avx-types.h:71:17: error: ‘__m512d’ does not name a type
ALIGNED __m512d ALIGNED d;
^
/pub/bin/GKL/src/main/native/pairhmm/avx-types.h:74:17: error: ‘__m512i’ does not name a type
ALIGNED __m512i ALIGNED i;
^
make[2]: *** [src/main/native/pairhmm/CMakeFiles/gkl_pairhmm.dir/avx_impl.cc.o] Error 1
make[1]: *** [src/main/native/pairhmm/CMakeFiles/gkl_pairhmm.dir/all] Error 2
make: *** [all] Error 2
:cmakeBuild FAILED

FAILURE: Build failed with an exception.

• What went wrong:
  Execution failed for task ':cmakeBuild'.

Process 'command 'make'' finished with non-zero exit value 2

• Try:
  Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

I have migrated gatk to arm architecture, but the running speed is too slow, so I want to use GKL. Can I build GKL on ARM architecture? Could you help me?

Hi Developer,

I was trying to accelerate Mutect2. I used class Thread to create new threads to parallelize the data prepare and active region calculation. However, after this modification, the calculation of pairHMM was much slower than before using the same data. I used the GKL source code with the commit id 'a6f5f4'.

Thank you very much!

IntelDeflaterFactory does not load the GKL until makeDeflater is called. This is too late because I want to know what deflater will be used before any deflater is actually created. See https://github.com/broadinstitute/gatk/tree/ak_use_intelDeflaterFactory and the workaround i put in CommandLineProgram for this issue.

Restore the AVX check in IntelGKLUtils.load from GKL 0.2.0.

For broadinstitute/gatk#2315

Memory at uninitialized pointer is being written to here

Remove factor of 3 in the total cell calculation to match the total cell calculation in other published PairHMM implementations.

Change this line to this:

m_total_cells = total_read_length * total_hap_length;

Match the GATK Smith-Waterman API and results.

For broadinstitute/gatk#1812

As reported by @kachulis in broadinstitute/gatk#8141, we are finding that the IntelDeflater in GKL version 0.8.8 seems to intermittently fail to properly compress outputs:

I run PrintReads over and over again, on the same input data, not doing anything, just read in, write out, ie gatk PrintRead -I input.bam -O output.bam. Mostly, I just get an identical 9GB bam over and over again (as confirmed by md5). However, sometimes (~10% of the time it seems), I get a MUCH larger “bam”, more like ~45GB. In runs where I get these larger output files, they are not always the same size, sometimes 45GB, sometimes 47GB (still always with the same input file, same commandline, same wdl task, etc). The runs that produce these larger bam also take much longer, with slower “reads per minute rate). They report exactly the same number of reads processed in the logs as the “normal” runs.

Looking inside the large output “bams” with gsutil cat, I see the header suddenly transitioning from compressed looking jibberish to a plaintext header, and then after a bit back to compressed looking jibberish again. Additionally, if I run these large bams through samtools view to get samtools to write them as a bam (ie samtools view big.bam -o samtools_out.bam) the resulting bam is much smaller ~6GB. It kind of seems like sometimes gatk will just stop compressing the output, and then start back up again, seemingly randomly??

This does not occur with GKL 0.8.6, and seems to have been introduced by the upgrade from GKL 0.8.6 to GKL 0.8.8 in https://github.com/broadinstitute/gatk/pull/7203/files

Additional data points:

• Reproducible on very small files (at about same rate of ~10%)
• Appears to be related to the IntelDeflater. when running with JDK deflater (--use-jdk-deflater) all 100/100 runs result in same sized bam

Any help would be much appreciated! This is actually a rather serious issue for us that might force us to temporarily revert back to the JDK deflater or the older GKL release if it looks like it might be difficult to diagnose / fix.

(CC @lbergelson)

I noticed that IntelInflator can issue a warning that says "Zero Bytes Written". This is confusing because it's happening during inflation when nothing is being written. I glanced at the class and it seems like there are a bunch of other probable copy/paste documentation errors where it looks like the deflator interface was copied and the text wasn't changed

The S/W aligner does not return optimal alignments.
Tested version: 0.8.6
Repo test case:

    @Test
    public void should_return_maximal_alignment_match() {
        IntelSmithWaterman isw = new IntelSmithWaterman();
        assertTrue(isw.load(null));
        SWNativeAlignerResult alignment = isw.align(
                "GGGGGGTTTTTT".getBytes(),
                "AAACCCTTTTTT".getBytes(),
                // BWA alignment scoring matrix
                new SWParameters(1, -4, -6, -1),
                SWOverhangStrategy.SOFTCLIP);
        assertEquals("6S6M", alignment.cigar);
        assertEquals(6, alignment.alignment_offset);
    }

Actual alignment cigar returned is 11S1M. This is clearly a sub-optimal alignment.

I strongly recommend adding a test suite to exhaustively compare small results against another S/W aligner (e.g. JAligner) to ensure correctness of your implementation.

Could you add a calculateDouble() stub function for double precision FPGA implementations ?

This would allow FPGA developers to create FPGA double precision implementation that are compatible with the GKL. Since none of the other interfaces change, this change will not affect any other software using GKL.

This would require :

• Adding the stub in the shacc_pairhmm namespace
• Change the allocation of the result table in JavaData.h (line 105, 106) to double (this would still work with the float version but use 2x the memory) or create a specific getBatchDouble() function.

// allocate results
m_batch.results = (float*)malloc(sizeof(float) * num_testcases); // Change to (double*)

• Change the if (g_use_fpga && !g_use_double) in IntelPairHmm.cc to

if (g_use_fpga) {
    if (g_use_double) {
        batch = javaData.getBatchDouble();
        shacc_pairhmm::calculate_double()
    } else {
        batch = javaData.getBatch();
        shacc_pairhmm::calculate()
    }
}

With this, if the GKL is called with the use_double and use_fpga option set to true it would call the new stub or if linked with a library that calls a FPGA implementation use the FPGA implementation.

Edit : Lines 74-76 in IntelPairHmm.java should also be changed.

if (args.useDoublePrecision && useFpga) {
    logger.warn("FPGA PairHMM does not support double precision floating-point. Using AVX PairHMM");
}

Could I get a consult on GATK user questions at https://gatkforums.broadinstitute.org/gatk/discussion/10353/gatk-3-8-0-printreads-fatal-error#latest?

Users are experiencing issues for v3.8 of GATK, which implements the GKL by default. I'm not sure if the issues are related to GKL, and if they are what are the guidelines we should provide users experiencing such issues.

Thanks.

It looks like a GATK user is experiencing a GKL hang since they updated their kernel (4.4 -> 4.12, SUSE Linux, SLES15, CPUs: Xeon E5-2697 v4 and Xeon Gold 6148 CPUs). They detailed their findings in a forum post:

https://gatkforums.broadinstitute.org/gatk/discussion/23342/inteldeflater-problem-on-kernel-update/p1

@gspowley - Can you take a look and see if you can make more sense of it?

Maybe it can catch some of these pointer safety issues reported in other tickets...

See test case in: broadinstitute/gatk#6733

Generate JNI header files as needed using CMake.

Hello,

Is the FPGA-acceleration OpenCL code open-source?

Thanks,
Tom

Please remove the fprintf on this line before the next release.

I was mentioning that there was some analysis showing other libraries outperforming the gkl on compression of genomics data. Here's the link http://www.htslib.org/benchmarks/zlib.html.

It looks like libdeflate seems to outperform the current intel zlib implementation in most metrics. I don't know if anything can be done to close the gap?

I wanted to open an issue here with some examples of crashes in the native accelerated smith waterman code.

See this GATK issue here: broadinstitute/gatk#5690

Two users encountered different the following issues which resolved by not using the accelerated code:

*** Error in `java': munmap_chunk(): invalid pointer: 0x00007fafc8ed1000 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fafce3f37e5]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x1a8)[0x7fafce400698]
/home/flexray/germline/cromwell-executions/PairedEndSingleSampleWorkflow/af1ee082-8661-4a7a-adf9-1b2a67333d37/call-HaplotypeCaller/shard-40/tmp.42584bbe/libgkl_smithwaterman205796788520033039.so(_Z19runSWOnePairBT_avx2iiiiPhS_iiaPcPs+0x338)[0x7faf73bfcfa8]
/home/flexray/germline/cromwell-executions/PairedEndSingleSampleWorkflow/af1ee082-8661-4a7a-adf9-1b2a67333d37/call-HaplotypeCaller/shard-40/tmp.42584bbe/libgkl_smithwaterman205796788520033039.so(Java_com_intel_gkl_smithwaterman_IntelSmithWaterman_alignNative+0xd8)[0x7faf73bfcbf8]
[0x7fafb9a7eea2]

and

*** Error in `java': double free or corruption (out): 0x00007f933d610780 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f93434427e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7f934344b37a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f934344f53c]
/home/flexray/germline/cromwell-executions/PairedEndSingleSampleWorkflow/fa8e6a15-021e-48cc-9429-c53596fc9c29/call-HaplotypeCaller/shard-19/tmp.ea81c1bd/libgkl_smithwaterman4419442010051805328.so(_Z19runSWOnePairBT_avx2iiiiPhS_iiaPcPs+0x338)[0x7f9248e4bfa8]
/home/flexray/germline/cromwell-executions/PairedEndSingleSampleWorkflow/fa8e6a15-021e-48cc-9429-c53596fc9c29/call-HaplotypeCaller/shard-19/tmp.ea81c1bd/libgkl_smithwaterman4419442010051805328.so(Java_com_intel_gkl_smithwaterman_IntelSmithWaterman_alignNative+0xd8)[0x7f9248e4bbf8]
[0x7f932de9ceaa]

I haven't reproduced either myself so I don't have a great way to test for them. It's possible we could recontact one of the original reporters but it's a been a long time.

I find the whole source code. can't find the impl. Could you help me?