slurm vs sge about somaticseq HOT 12 CLOSED

Thanks for the prompt reply. How do I pass in other bamsurgeon options, such as –ignorepileup Xiaopeng From: Li Tai Fang <[email protected]> Sent: Thursday, December 12, 2019 1:07 AM To: bioinform/somaticseq <[email protected]> Cc: Bian, Xiaopeng (NIH/NCI) [E] <[email protected]>; Author <[email protected]> Subject: Re: [bioinform/somaticseq] slurm vs sge (#75) As long as the files are being created, those error messages are just for variants in the random_.bed file that don't make it into the final synthetic_.vcf files (i.e., variants attempted but did not make into the final synthetic bam file). For details about bamsurgeon, you should contact its author https://github.com/adamewing/bamsurgeon. I've just created some run scripts for it. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#75?email_source=notifications&email_token=ABRXDTLL4UEZVFM5PEQ7QFTQYHIJTA5CNFSM4JZZAEX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGVSK4I#issuecomment-564864369>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABRXDTNU3MKJRQPTC3HD5C3QYHIJTANCNFSM4JZZAEXQ>.

Hi, I have a question about bamsimulator scenario #1 An ideal example is when you have sequencing replicates of the same normal samples Is this the same as if I use two copy of the same normal bam, one named “normal”, the other named “tumor”? Thanks. Xiaopeng Bian

Hi, Thanks for the explanation! I followed the example three for multi-thread jobs that merge and then split the input tumor and normal bam files. All ran well except that I had to run the final merge commands manually to merge bam and vcf files. When I use generated bam and vcf files and ran the training mode in Neurosomatic, I got a “FileNotFoundError: [Errno 2] No such file or directory: 'work_train/dataset/*/candidates*.tsv.idx'” error. Would you please have a look at the attached log file and let me know what are the possible reason why the *tsv.idx did not generated and give me some suggestions how to fix it? Thanks. Xiaopeng From: Li Tai Fang <[email protected]> Sent: Thursday, January 23, 2020 8:05 PM To: bioinform/somaticseq <[email protected]> Cc: Bian, Xiaopeng (NIH/NCI) [E] <[email protected]>; Author <[email protected]> Subject: Re: [bioinform/somaticseq] slurm vs sge (#75) Not really. If you sequence the same genome twice, there will be differences between them, including sequencing errors and different reads. Some of the differences will be called as mutations (false positives), so the machine learning model can learn them. If the same bam file is used, there is no difference between them, so there will be no false positive call. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#75?email_source=notifications&email_token=ABRXDTM4IMWDS5RHVM5FMFTQ7I5CJA5CNFSM4JZZAEX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJZM5ZY#issuecomment-577949415>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABRXDTJYG3ZTTJKH5V7A3QLQ7I5CJANCNFSM4JZZAEXQ>.

Hi, I ran the Bamsimulator multi thread on whole exome data, so I only spike in snv, no sv or indel, but I saw a small (9k) “synthetic_indels.leftAlign.vcf” in the final results. Because the merge commands did not run correctly, I had to run the merge commands in the log dir after all threads finished, that should be fine? I just want to make sure the Bamsimulator ran correctly so the problems I had is in the model training step. Thanks. Xiaopeng From: Li Tai Fang <[email protected]> Sent: Thursday, January 23, 2020 8:05 PM To: bioinform/somaticseq <[email protected]> Cc: Bian, Xiaopeng (NIH/NCI) [E] <[email protected]>; Author <[email protected]> Subject: Re: [bioinform/somaticseq] slurm vs sge (#75) Not really. If you sequence the same genome twice, there will be differences between them, including sequencing errors and different reads. Some of the differences will be called as mutations (false positives), so the machine learning model can learn them. If the same bam file is used, there is no difference between them, so there will be no false positive call. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#75?email_source=notifications&email_token=ABRXDTM4IMWDS5RHVM5FMFTQ7I5CJA5CNFSM4JZZAEX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJZM5ZY#issuecomment-577949415>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABRXDTJYG3ZTTJKH5V7A3QLQ7I5CJANCNFSM4JZZAEXQ>.

Thank you very much for your help! I am trying to create a synthetic tumor/normal pair from a pair of hapmap WEX tumor/normal samples with ground truth. I’d like to use them in training a Neusomatic model for SNV calling, in hope that it will improve performance over some of the current popular variant callers, such as Mutect2, MuSE, LoFreq, etc. I’ve learned that Bamsurgeon is not optimal for WEX data, especially for SV spike ins, but that is not what I am interested now. I followed your example 3 script (attached) created the synthetic pair and Neusomatic script (also attached) to train a model used in Neusomatic calling. But did not get better result than other callers. I think it could be improved by modifying parameters. Would you have a look at the attached script and give me some suggestions which parameters I should change/add to improve the calling performance? Or if you can point me to any online resources where I can get some information. Thanks again! Xiaopeng From: Li Tai Fang <[email protected]> Sent: Thursday, February 6, 2020 2:07 PM To: bioinform/somaticseq <[email protected]> Cc: Bian, Xiaopeng (NIH/NCI) [E] <[email protected]>; Author <[email protected]> Subject: Re: [bioinform/somaticseq] slurm vs sge (#75) Sounds like things are fine. If the synthetic_XXX.vcf are not empty, it means those variants were created in the reads. In the .sh script, there is a default number of indels. If you want no indel, you would have to explicitly have the parameter --num-indels 0. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#75?email_source=notifications&email_token=ABRXDTPPFDSSQDVRJTM2W6LRBRNVRA5CNFSM4JZZAEX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELAM5JA#issuecomment-583061156>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABRXDTMPQLLUXMTWG7RTAV3RBRNVRANCNFSM4JZZAEXQ>. #!/bin/sh module load somaticseq module load singularity /usr/local/apps/somaticseq/3.3.0/utilities/singularities/bamSimulator/BamSimulator_multiThreads.sh \ --output-dir /gpfs/gsfs4/users/nextgen/Xiaopeng/match/neusomatic/MATCH/bamsimulator/hapmap_output_multi_mda_snv \ --genome-reference /gpfs/gsfs4/users/nextgen/Xiaopeng/match/neusomatic/MATCH/ref/hg38.fa \ --tumor-bam-in /gpfs/gsfs4/users/nextgen/Xiaopeng/match/neusomatic/MATCH/bamsimulator/hapmap/mda/R1-XX.bam \ --normal-bam-in /gpfs/gsfs4/users/nextgen/Xiaopeng/match/neusomatic/MATCH/bamsimulator/hapmap/mda/R1-GM07014.bam \ --tumor-bam-out syntheticTumor.bam \ --normal-bam-out syntheticNormal.bam \ --selector /data/nextgen/Xiaopeng/match/neusomatic/MATCH/ref/broad_MDA_mocha_overlap_cds.bed \ --split-proportion 0.5 \ --min-variant-reads 2 \ --threads 24 \ --action "sbatch --time=60:00:00" \ --num-snvs 30000 \ --min-vaf 0.03 --max-vaf 1.0 --left-beta 2 --right-beta 5 \ --merge-bam --split-bam --merge-output-bams #!/bin/bash module load neusomatic refGenome=/gpfs/gsfs4/users/nextgen/Xiaopeng/match/neusomatic/MATCH/ref/hg38.fa bed=/data/nextgen/Xiaopeng/match/neusomatic/MATCH/ref/broad_MDA_mocha_overlap_cds.bed input_dir=/data/nextgen/Xiaopeng/match/neusomatic/MATCH/bamsimulator/hapmap_output_multi_mda_snv echo "scan Align, ${NEUSOMATIC_SCAN_ALIGNMENTS}" preprocess.py \ --mode train \ --reference ${refGenome} \ --region_bed ${bed} \ --tumor_bam ${input_dir}/syntheticTumor.bam \ --normal_bam ${input_dir}/syntheticNormal.bam \ --work ${input_dir}/work_train_mergedVCF \ --truth_vcf ${input_dir}/synthetic_merged.vcf \ --min_mapq 10 \ --num_threads 8 \ --scan_alignments_binary ${NEUSOMATIC_SCAN_ALIGNMENTS} train.py \ --candidates_tsv ${input_dir}/work_train_mergedVCF/dataset/*/candidates*.tsv \ --out ${input_dir}/work_train_mergedVCF \ --num_threads 8 \ --batch_size 100

Hi, Thank you very much for your suggestions! Yes, I use “broad_MDA_mocha_overlap_cds.bed” as the regions for spliking SNVs into a pair of hapmap WEX tumot/normal files following your Example 3 multi thread process at: https://github.com/bioinform/somaticseq/tree/master/utilities/dockered_pipelines/bamSimulator I understand the idea of your suggestion but I don’t know to actually do it. 1. You suggest “spike in a few thousand into each pair of synthetic WEX data sets”, you actually mean the real data sets I am using, is that right? 2. You suggest “increase the number of synthetic BAM files”, I have four pair of samples, so I could spike in say 8000 SNVs into each of them to generated 4 sets of synthetic tumor/normal pairs? 3. “to have enough training data”, how do I feed the multi synthetic data sets generated above into Neosomatic training process? Merge them into a single set first? Thanks very much! Xiaopeng From: Li Tai Fang <[email protected]> Sent: Sunday, February 9, 2020 10:33 AM To: bioinform/somaticseq <[email protected]> Cc: Bian, Xiaopeng (NIH/NCI) [E] <[email protected]>; Author <[email protected]> Subject: Re: [bioinform/somaticseq] slurm vs sge (#75) Wondering if the bam files you used to spike SNVs are WXS files with broad_MDA_mocha_overlap_cds.bed as the regions? If you use a single pair of WEX while spiking in 30,000 SNVs, the synthetic tumor-normal pairs aren't going to have the "right ratio of true positives and false positives." I think a more optimal way is to spike in a few thousand into each pair of synthetic WEX data sets, but increase the number of synthetic BAM files, in order to have enough true positives but also enough false positives to have enough training data. For details of neusomatic, contact https://github.com/bioinform/neusomatic. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#75?email_source=notifications&email_token=ABRXDTMYVX4OOJ5MZVHC25TRCAO2LA5CNFSM4JZZAEX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELGPP4A#issuecomment-583858160>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABRXDTJF7YL2PONPLEHDEWTRCAO2LANCNFSM4JZZAEXQ>.