FixMateInformation crashes on supplemental alignments where the mate is missing about picard HOT 14 CLOSED

The symptom can be created by setting the supplementary alignment flag for the first segment of a pair.
A simple contrived example:

>java -jar picard.jar ViewSam I=issue161example.bam
@HD VN:1.5  SO:coordinate
@SQ SN:xx   LN:100
@RG ID:grp1 PL:ILLUMINA LB:Library 1    DS:Group 1  SM:Sample
a1  2179    xx  1   1   10M =   41  50  AAAAAAAAAA  **********  MC:Z:10M    RG:Z:grp1   NM:i:0  MQ:i:1
a1  147 xx  41  1   10M =   1   -50 TTTTTTTTTT  **********  MC:Z:10M    RG:Z:grp1   NM:i:0  MQ:i:1
[Sat Sep 26 15:28:51 PDT 2015] picard.sam.ViewSam done. Elapsed time: 0.01 minutes.

>java -jar picard.jar ValidateSamFile I=issue161example.bam
ERROR: Read name a1, Mate not found for paired read
[Sat Sep 26 15:38:13 PDT 2015] picard.sam.ValidateSamFile done. Elapsed time: 0.01 minutes.

>java -jar picard.jar FixMateInformation I=issue161example.bam O=fixed.bam
... Picard version: 1.139(8ceee52414e8ab9d13e350ff9cd86d48825dd64d_1442240108) JdkDeflater
INFO    2015-09-26 15:42:20 FixMateInformation  Sorting input into query name order.
INFO    2015-09-26 15:42:20 FixMateInformation  Sorting by queryname complete.
INFO    2015-09-26 15:42:20 FixMateInformation  Output will be sorted by coordinate
INFO    2015-09-26 15:42:20 FixMateInformation  Traversing query name sorted records and fixing up mate pair information.
[Sat Sep 26 15:42:20 PDT 2015] picard.sam.FixMateInformation done. Elapsed time: 0.01 minutes.
Runtime.totalMemory()=16121856
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" java.lang.NullPointerException
    at htsjdk.samtools.SamPairUtil.setMateInformationOnSupplementalAlignment(SamPairUtil.java:300)
    at htsjdk.samtools.SamPairUtil$SetMateInfoIterator.advance(SamPairUtil.java:445)
    at htsjdk.samtools.SamPairUtil$SetMateInfoIterator.next(SamPairUtil.java:454)
    at htsjdk.samtools.SamPairUtil$SetMateInfoIterator.next(SamPairUtil.java:360)
    at picard.sam.FixMateInformation.doWork(FixMateInformation.java:195)
    at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:206)
    at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
    at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)

from picard.

A simple solution is to subordinate supplementary-alignment processing, i.e.

            // Set mate information on supplemental records
            if (containsSupplementalRecord) {
              ...
            }

to the condition (immediately preceding) that both pair-alignments have been identified -

            // we must find both records to update the mate info
            if (null != firstPrimaryRecord && null != secondPrimaryRecord) {

in SamPairUtil.java advance()

from picard.

I think it is odd that a tool that is trying to fix mate information should be OK with the case where one end of a pair is missing. Nonetheless, this is the current behavior, so I added an option to allow for the case where we want to catch this. This requires a htsjdk and picard update.

from picard.

Is it possible for FixMateInformation to fix the mate information for a read for which no mate is found? That is, strip mate information and remove SAM flags indicating that it is paired.

It seems strange (and contrary to the usage I was just attempting) that after running FixMateInformation, there would still be reads with incorrect mate information left.

from picard.

The original issue has been fixed.

@d-cameron Sorry for the late response. I think there are indeed some things that FMI doesn't address but I don't know the details. @nh13 may be able to tell you more.

from picard.

I am using version 2.19, but I still have the same error while using FixMateInformation. Has it been implemented in picard's newer versions?
If not, where can I find SamPairUtil.java to make the changes in pair alignment detection as mentioned in previous threads?
Thanks!

from picard.

are you setting IGNORE_MISSING_MATES=true?

from picard.

Yes, I am.
The code-
$java -Xms20g -Xmx42g -jar /work/jigyasa-arora/picard.jar FixMateInformation I=272-23-original-sorted.bam O=272-23-original-sorted-fixedmate.bam ADD_MATE_CIGAR=true IGNORE_MISSING_MATES=true ASSUME_SORTED=true

The error-
FixMateInformation INPUT=[272-23-original-sorted.bam] OUTPUT=272-23-original-sorted-fixedmate.bam ASSUME_SORTED=true ADD_MATE_CIGAR=true IGNORE_MISSING_MATES=true VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
[Thu Apr 25 22:57:17 JST 2019] Executing as [email protected] on Linux 3.10.0-327.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_65-b17; Deflater: Intel; Inflater: Intel; Provider GCS is not available; Picard version: 2.19.2-SNAPSHOT
INFO 2019-04-25 22:57:45 FixMateInformation Output will be sorted by coordinate
INFO 2019-04-25 22:57:49 FixMateInformation Traversing query name sorted records and fixing up mate pair information.
[Thu Apr 25 22:57:49 JST 2019] picard.sam.FixMateInformation done. Elapsed time: 0.54 minutes.
Runtime.totalMemory()=25949634560

from picard.

I don't see the error in the output you posted...

from picard.

sorry, here goes the whole output-
[Thu Apr 25 22:57:17 JST 2019] FixMateInformation INPUT=[272-23-original-sorted.bam] OUTPUT=272-23-original-sorted-fixedmate.bam ASSUME_SORTED=true ADD_MATE_CIGAR=true IGNORE_MISSING_MATES=true VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
[Thu Apr 25 22:57:17 JST 2019] Executing as [email protected] on Linux 3.10.0-327.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_65-b17; Deflater: Intel; Inflater: Intel; Provider GCS is not available; Picard version: 2.19.2-SNAPSHOT
INFO 2019-04-25 22:57:45 FixMateInformation Output will be sorted by coordinate
INFO 2019-04-25 22:57:49 FixMateInformation Traversing query name sorted records and fixing up mate pair information.
[Thu Apr 25 22:57:49 JST 2019] picard.sam.FixMateInformation done. Elapsed time: 0.54 minutes.
Runtime.totalMemory()=25949634560
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" htsjdk.samtools.SAMException: Found two records that are paired, not supplementary, and first of the pair
at htsjdk.samtools.SamPairUtil$SetMateInfoIterator.advance(SamPairUtil.java:454)
at htsjdk.samtools.SamPairUtil$SetMateInfoIterator.next(SamPairUtil.java:501)
at htsjdk.samtools.SamPairUtil$SetMateInfoIterator.next(SamPairUtil.java:388)
at picard.sam.FixMateInformation.doWork(FixMateInformation.java:228)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:295)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:103)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:113)

from picard.

I think that this is a different issue. your file has two primary (i.e. not secondary or supplementary) reads with the same name but are both marked "first in pair". this cannot be fixed with this tool.

from picard.

(what can be fixed is that the tool can be modified to show the readname on this error.)

from picard.

samtools/htsjdk#1358

from picard.

I think that this is a different issue. your file has two primary (i.e. not secondary or supplementary) reads with the same name but are both marked "first in pair". this cannot be fixed with this tool.

i've just encountered the same error, is there a best practice method to fix this?

from picard.