dnanexus / wdltools Goto Github PK

View Code? Open in Web Editor NEW

23.0 23.0 7.0 2.56 MB

WDL tools for parsing, type-checking, and more

License: Apache License 2.0

Makefile 0.08% Scala 88.59% ANTLR 3.14% WDL 8.17% Shell 0.02%

wdltools's Introduction

DNAnexus

Dnanexus Apps and Scripts

applets

binning_step0: BioBin Pipeline
biobin_pipeline
binning_step1: BioBin Pipeline
biobin_pipeline
binning_step2: BioBin Pipeline
biobin_pipeline
binning_step3: BioBin Pipeline
biobin_pipeline
impute2_group_join: Impute2_group_join
This app can be used to merge multiple imputed impute2 files
plato_biobin: PLATO BioBin Regression Analysis
PLATO_BioBin
vcf_batch: VCF Batch effect tester
vcf_batch

apps

association_result_annotation: Annotate GWAS, PheWAS Assocaitions
association_result_annotation
biobin:
This app runs the latest development build of the rare variant binning tool BioBin.
generate_phenotype_matrix: Generate Phenotype Matrix
generate_phenotype_matrix
genotype_case_control: Generate Case/Control by Genotype
App provides case and control number by each genotype
impute2: imputation
This will perfrom imputation using Impute2
impute2_to_plink: Impute2 To PLINK
Convert Impute2 file to PLINK files
plato_single_variant: PLATO - Single Variant Analysis
Apps allows you to run single variant association testing against single phenotype (GWAS) or multiple phenotype (PheWAS) test
rl_sleeper_app: sleeper
This App provides some useful tools when working with data in DNANexus. This App is designed to be run on the command line with "dx run --ssh RL_Sleeper_App" in the project that you have data that you want to explore (use "dx select" to switch projects as needed).
shapeit2: SHAPEIT2
This app do phasing using SHAPEIT2
strand_align: Strand Align
Strand Align prior to phasing
vcf_annotation_formatter:
Extracts and reformats VCF annotations (CLINVAR, dbNSFP, SIFT, SNPEff)
QC_apps subfolder:
- drop_marker_sample: Drop Markers and/or Samples (PLINK)
  - drop_marker_sample
drop_relateds: Relatedness Filter (IBD)
- drop_relateds
extract_marker_sample: Drop Markers and/or Samples (PLINK)"
- extract_marker_sample
maf_filter: Marker MAF Rate Filter (PLINK)
- maf_filter
marker_call_filter: Marker Call Rate Filter (PLINK)
- marker_call_filter
missing_summary: Missingness Summary (PLINK)
- Returns missingness rate by sample
pca: Principal Component Analysis using SMARTPCA
- pca
sample_call_filter: Sample Call Rate Filter (PLINK)
- sample_call_filter

scripts

cat_vcf.py *
download_intervals.py *
download_part.py *
estimate_size.py *
interval_pad.py
- This reads a bed file from standard input, pads the intervals, sorts and then outputs the intervals guranteed to be non-overlapping
update_applet.sh *

sequencing

bcftools_view:
- Calls "bcftools view". Still in experimental stages.
calc_ibd:
- Calculates a pairwise IBD estimate from either VCF or PLINK files using PLINK 1.9.
call_bqsr: Base Quality Score Recalibration
- Call GATK BaseRecalibrator and return the tables for use in HaplotypeCaller
call_genotypes:
- Obsolete, do not use; use geno_p instead. Calls GATK GenotypeGVCFs.
call_hc:
- Call GATK HaplotypeCaller and return gVCF files
call_vqsr:
- Calls GATK VariantRecalibrator and returns the files needed to apply the recalibration
cat_variants: combine_variants
- Combines non-overlapping VCF files with the same subjects. A reimplementation of GATK CatVariants (GATK CatVariants available upon request)
combine_variants: combine_variants
- Calls GATK CombineVariants to merge VCF files
gen_ancestry:
- Determine Ancestry from PCA. Uses an eigenvector file and training dataset listing known ancestries. Runs QDA to determine posterior ancestries for all samples, even those in the training set.
gen_related_todrop:
- Uses a PLINK IBD file to determine the minimal set of samples to drop in order to generate an unrelated sample set. Uses a minimum vertex cut algorithm of the related samples to get
geno_p:
- Calls GATK GenotypeGVCFs in parallel by chromosome
merge_gvcfs:
- Calls GATK CombineGVCFs
plink_merge:
- Merge PLINK bed/bim/fam files using PLINK 1.9
select_variants: VCF QC
- Calls GATK SelectVariants
variant_annotator: VCF QC
- Calls GATK VariantAnnotator
vcf_annotate: Annotate VCF File
- Use a variety of tools to annotate a sites-only VCF.
vcf_concordance: VCF Concordance
- Generate concordance metrics from VCF file(s) using GATK GenotypeConcordance. Not recommended for large files.
vcf_gen_lof:
- Subset a VCF from vcf_annotate based on the given annotations to get a sites-only VCF of loss-of-function variants.
vcf_pca:
- Uses PLINK 1.9 and eigenstrat 6.0 to calculate principal components from VCF or PLINK bed/bim/fam files.
vcf_qc:
- Calls GATK ApplyRecalibration and GATK VariantFiltration to apply filters to VCF files.
vcf_query:
- Calls "bcftools query" to extract annotations from the VCF file. Used in the stripping of files for MEGAbase
vcf_sitesonly: VCF QC
- Generates a sites-only file from full VCF files.
vcf_slice: Slice VCF File(s)
- Return a small section of a VCF file (similar to tabix). For large output, many small regions, or subsetting samples, use subset_vcf instead.
vcf_summary: VCF Summary Statistics
- Generate summary statistics for a VCF file (by sample and by variant)
vcf_to_plink:
- Uses PLINK 1.9 to convert VCF files to PLINK bed/bim/fam files

wdltools's People

Contributors

Stargazers

Watchers

Forkers

commandlinegirl rhpvorderman mhrvol scalavision wook2014 yangy369 jdidion

wdltools's Issues

Add comments to AST

format subcommand throws error when file ends in a comment

$ wdltools --version
wdlTools 0.17.8
$ cat test.wdl
version 1.0
workflow my_workflow {
  input { String message }
	call display { input: message=message }
}

task display {
  input { String message }
  command { printf '~{message}' }
}
# comment
$ wdltools check test.wdl
$ wdltools format test.wdl
[error] Command scallop failed
java.lang.IllegalArgumentException: requirement failed
	at scala.Predef$.require(Predef.scala:324)
	at wdlTools.generators.code.WdlFormatter$LineFormatter.endLine(WdlFormatter.scala:205)
	at wdlTools.generators.code.WdlFormatter$LineFormatter.maybeAppendFullLineComments(WdlFormatter.scala:311)
	at wdlTools.generators.code.WdlFormatter$LineFormatter.endSection(WdlFormatter.scala:173)
	at wdlTools.generators.code.WdlFormatter$DocumentSections.format(WdlFormatter.scala:1885)
	at wdlTools.generators.code.WdlFormatter.formatElement(WdlFormatter.scala:1898)
	at wdlTools.generators.code.WdlFormatter.formatDocument(WdlFormatter.scala:1907)
	at wdlTools.generators.code.WdlFormatter.$anonfun$formatDocuments$1(WdlFormatter.scala:1914)
	at wdlTools.syntax.WdlParser$Walker.addDocument$1(WdlParser.scala:66)
	at wdlTools.syntax.WdlParser$Walker.walk(WdlParser.scala:80)
	at wdlTools.generators.code.WdlFormatter.formatDocuments(WdlFormatter.scala:1913)
	at wdlTools.cli.Format.apply(Format.scala:15)
	at wdlTools.cli.Main$.runCommand(Main.scala:36)
	at wdlTools.cli.Main$.delayedEndpoint$wdlTools$cli$Main$1(Main.scala:48)
	at wdlTools.cli.Main$delayedInit$body.apply(Main.scala:12)
	at scala.Function0.apply$mcV$sp(Function0.scala:39)
	at scala.Function0.apply$mcV$sp$(Function0.scala:39)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
	at scala.App.$anonfun$main$1(App.scala:76)
	at scala.App.$anonfun$main$1$adapted(App.scala:76)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:926)
	at scala.App.main(App.scala:76)
	at scala.App.main$(App.scala:74)
	at wdlTools.cli.Main$.main(Main.scala:12)
	at wdlTools.cli.Main.main(Main.scala)

$

If the comment line is removed, then the format command does not throw an exception and shows the reformated file as expected.

API calls to write out WDL files

I was informed that the wdlTools Scala API has provision for writing out WDL files from a Scala program which describe the workflow via Scala data/objects.

I wasn't able to find example/tutorial Scala code in this report which illustrate them.

Some pointer to them would be very much appreciated.

Thank you.

Adding wdltools to conda-forge channel for easy redistribution with miniconda/anaconda

conda-forge/staged-recipes#12233

Dear @jdidion and @orodeh ,

Thanks for the great work on wdTools. I packaged wdlTools as the wdltools conda package. This makes distribution, installation and use much easier. i.e. conda install wdltools; wdltools <command> instead of wget <link to wdltools.jar>; java -jar <location of jar> <command>. Especially remembering the location of the jar every time is not nice. This makes updating painful.

With conda this problem is solved. I have added you both as maintainers for the package. Updating is easy: a pushed tag should be automatically detected. An auto-generated PR will follow. This makes stuff easy to update. Cromwell and miniwdl are already packaged on conda-forge. I packaged miniwdl, and maintenance has been so easy there was no involvement from me needed since.

If you don't want to maintain the package I will remove you from the list and I will maintain the package for you. If you are interested in maintaining it (i.e. you can determine when a new update is pushed to conda-forge) please state so in the above PR. It is required for the PR to be merged.

Thanks again!

90% test coverage

Allow "format" to read STDIN

I would love for the "format" command to accept input on STDIN so I can integrate this with my editor.

Use of reserved keyword as name of input/output results in an uninformative error message

Hello

I have encountered the following problem:

I have a workflow that contains (among others) the two tasks GetBwaVersion and MergeBamAlignment.
MergeBamAlignment takes as an input the output of GetBwaVersion.

task GetBwaVersion{
  input{
  String docker_image
  String bwa_path
  }
  command {
    # Not setting "set -o pipefail" here because /bwa has a rc=1 and we don't want to allow rc=1 to succeed
    # because the sed may also fail with that error and that is something we actually want to fail on.
    ~{bwa_path}bwa 2>&1 | \
    grep -e '^Version' | \
    sed 's/Version: //'
  }
  runtime {
    docker: docker_image
    dx_instance_type: "mem1_ssd1_v2_x2"
  }
  output {
    **String bwa_version = read_string(stdout())**
  }
}

  call MergeBamAlignment {
      input:
        unmapped_bam = FastqToUnmappedBam.unmapped_bam,
        bwa_commandline = bwa_commandline,
        index_prefix = SamToFastqAndBwaMem.bwa_index_prefix,
        **bwa_version = GetBwaVersion.version,**
        aligned_bam = SamToFastqAndBwaMem.output_bam,
        output_bam_basename = base_file_name + ".aligned.unsorted",
        ref_fasta = ref_fasta,
        ref_fasta_index = ref_fasta_index,
        ref_dict = ref_dict,
        docker_image = gatk_docker,
        gatk_path = gatk_path,
        compression_level = compression_level
    }

}

When I attempted to compile this workflow I got the following error message:

wdlTools.syntax.SyntaxException: error parsing document /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
Caused by: wdlTools.syntax.SyntaxException: missing identifier at 122:4-122:25 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl

As it turned out (after considerable frustration), the cause of this error is that version is apparently a reserved keyword in WDL, and thus naming the output of GetBwaVersion version resulted in the parser failing to find an identifier for this output.

I see three issues that make this difficult to debug:
1. Uninformative error message: The way this error message is phrased, it is very difficult to infer that the problem is the use of a reserved keyword. It would have been very helpful had the parser indicated this explicitly.
2. Lacunae in the documentation: I was unable to find any mention of the fact that version is a reserved keyword. It does not appear in the list of reserved keywords in the WDL specfications. I infer that it is a reserved keyword based on the fact that replacing the output name from version to bwa_version resulted in a successful compilation of the workflow, and also from the following clause in the wdlTools code:

it should "version is a reserved keyword" in {
    assertThrows[Exception] {
      val _ = getDocument(getWorkflowSource("call_bug2.wdl"))
    }

(source: https://github.com/dnanexus-rnd/wdlTools/blob/682efa4151c17dbeff627832d7f491df0436fc5f/src/test/scala/wdlTools/syntax/v1/ConcreteSyntaxV1Test.scala)
3. Inconsistent error messages between task and call: At same point I switched the name of the task output to bwa_version but forgot to change it in the call to MergeBamAlignment.
This resulted in the following cryptic error message:

wdlTools.syntax.SyntaxException: error parsing document /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
Caused by: wdlTools.syntax.SyntaxException: token recognition error at: ',' at  at 92:43-92:43 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
token recognition error at: '\n' at  at 92:44-92:44 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
token recognition error at: '_' at  at 93:15-93:15 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
no viable alternative at input ',\n        bwa_version = GetBwaVersion.version' at version at 92:36-92:36 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
extraneous input '=' expecting Identifier at = at 92:20-92:20 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
mismatched input '.' expecting '=' at . at 92:35-92:35 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
extraneous input '=' expecting Identifier at = at 93:20-93:20 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
mismatched input '.' expecting '=' at . at 93:41-93:41 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
extraneous input ',' expecting Identifier at , at 93:52-93:52 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
extraneous input ',' expecting {'scatter', 'call', 'if', 'input', 'output', 'parameter_meta', 'meta', 'Boolean', 'Int', 'Float', 'String', 'File', 'Array', 'Map', 'Pair', 'Object', RBRACE, Identifier} at , at 94:66-94:66 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
extraneous input '=' expecting Identifier at = at 95:18-95:18 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
mismatched input ',' expecting {'scatter', 'call', 'if', 'input', 'output', 'parameter_meta', 'meta', 'Boolean', 'Int', 'Float', 'String', 'File', 'Array', 'Map', 'Pair', 'Object', RBRACE, '[', '<', '>', '>=', '<=', '==', '!=', '&&', '||', '*', '+', '-', '.', '/', '%', Identifier} at , at 95:29-95:29 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
extraneous input '=' expecting Identifier at = at 96:24-96:24 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
mismatched input ',' expecting {'scatter', 'call', 'if', 'input', 'output', 'parameter_meta', 'meta', 'Boolean', 'Int', 'Float', 'String', 'File', 'Array', 'Map', 'Pair', 'Object', RBRACE, '[', '<', '>', '>=', '<=', '==', '!=', '&&', '||', '*', '+', '-', '.', '/', '%', Identifier} at , at 96:41-96:41 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
extraneous input '=' expecting Identifier at = at 97:17-97:17 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
mismatched input ',' expecting {'scatter', 'call', 'if', 'input', 'output', 'parameter_meta', 'meta', 'Boolean', 'Int', 'Float', 'String', 'File', 'Array', 'Map', 'Pair', 'Object', RBRACE, '[', '<', '>', '>=', '<=', '==', '!=', '&&', '||', '*', '+', '-', '.', '/', '%', Identifier} at , at 97:27-97:27 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
extraneous input '=' expecting Identifier at = at 98:21-98:21 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
mismatched input ',' expecting {'scatter', 'call', 'if', 'input', 'output', 'parameter_meta', 'meta', 'Boolean', 'Int', 'Float', 'String', 'File', 'Array', 'Map', 'Pair', 'Object', RBRACE, '[', '<', '>', '>=', '<=', '==', '!=', '&&', '||', '*', '+', '-', '.', '/', '%', Identifier} at , at 98:34-98:34 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
extraneous input '=' expecting Identifier at = at 99:18-99:18 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
mismatched input ',' expecting {'scatter', 'call', 'if', 'input', 'output', 'parameter_meta', 'meta', 'Boolean', 'Int', 'Float', 'String', 'File', 'Array', 'Map', 'Pair', 'Object', RBRACE, '[', '<', '>', '>=', '<=', '==', '!=', '&&', '||', '*', '+', '-', '.', '/', '%', Identifier} at , at 99:29-99:29 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
extraneous input '=' expecting Identifier at = at 100:26-100:26 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
mismatched input '}' expecting {'scatter', 'call', 'if', 'input', 'output', 'parameter_meta', 'meta', 'Boolean', 'Int', 'Float', 'String', 'File', 'Array', 'Map', 'Pair', 'Object', RBRACE, '[', '<', '>', '>=', '<=', '==', '!=', '&&', '||', '*', '+', '-', '.', '/', '%', Identifier} at } at 101:4-101:4 in

The line numbers in the error message correspond to the call to MergeBamAlignment, but suggest a localized syntax problem such as a misplaced comma/newline and not an inconsistency between the name of the output of the task definition and its name in the call.
I assume that this is also caused by the use of a reserved keyword. To test this, I changed
the output name in the task to bwa_version and set the input of the call to
bwa_version = GetBwaVersion.bwa_version2
Which resulted in the much more informative error message:

wdlTools.types.TypeException: Call GetBwaVersion does not have output bwa_version2 at 92:22-92:35 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl
argument to parameter 'bwa_version' has type T_Call(GetBwaVersion,TreeSeqMap(bwa_version -> T_String)),  it is not coercible to T_String at 87:4-101:5 in /home/hadassah/Documents/Dolev/pipeline_automation/sandbox/mva_dxcompiler.wdl

Build document graph

Add to the Graph module the ability to build a hierarchical document graph from a document and (recursively) all its imports.

It would be useful to have this at both the AST and TST level. The AST-level graph would be particularly useful as the basis for type inference. It may be easiest to build the TST-level graph during type inference.

Parser bug when upgrading from draft-2 to v1.1

Hi, this tool is great, and I appreciate the work that's gone into this, but I'm running into an issue when I use the wdlTools upgrade feature. I was wondering what I could do to help resolve this?

I suspect that the issue has to do with the parser looking for a version regardless of whether the document is draft-2 or not

Thanks so much for this excellent tool!

I ran the command like this:

java -jar ../wdlTools-0.17.4.jar upgrade -f -s draft-2 -d 1.1 -o -O test_overwrite FastQC/fastqc.wdl

I'm on macOS Monterey 12.0.1 (x86_64) Java:

openjdk 11.0.12 2021-07-20 LTS
OpenJDK Runtime Environment Zulu11.50+19-CA (build 11.0.12+7-LTS)
OpenJDK 64-Bit Server VM Zulu11.50+19-CA (build 11.0.12+7-LTS, mixed mode)

The error that I'm getting looks like this

Upgrading FastQC/fastqc.wdl
[error] Command scallop failed
java.lang.Exception: Failed to parse FastQC/fastqc.wdl; this is likely due to a bug in the WDL code generator - please report this issue
        at wdlTools.cli.Upgrade.parseAndCheck(Upgrade.scala:23)
        at wdlTools.cli.Upgrade.apply(Upgrade.scala:52)
        at wdlTools.cli.Main$.runCommand(Main.scala:36)
        at wdlTools.cli.Main$.delayedEndpoint$wdlTools$cli$Main$1(Main.scala:48)
        at wdlTools.cli.Main$delayedInit$body.apply(Main.scala:12)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1(App.scala:73)
        at scala.App.$anonfun$main$1$adapted(App.scala:73)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:553)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:551)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:920)
        at scala.App.main(App.scala:73)
        at scala.App.main$(App.scala:71)
        at wdlTools.cli.Main$.main(Main.scala:12)
        at wdlTools.cli.Main.main(Main.scala)
Caused by: wdlTools.syntax.SyntaxException: error parsing document FastQC/fastqc.wdl
        at wdlTools.syntax.v1_1.ParseAll.parseDocument(ParseAll.scala:402)
        at wdlTools.cli.Upgrade.parseAndCheck(Upgrade.scala:20)
        ... 16 more
Caused by: java.lang.Exception: unrecognized version
        at wdlTools.syntax.v1_1.ParseTop.visitVersion(ParseTop.scala:1376)
        at wdlTools.syntax.v1_1.ParseTop.visitDocument(ParseTop.scala:1388)
        at wdlTools.syntax.v1_1.ParseTop.$anonfun$parseDocument$1(ParseTop.scala:1420)
        at wdlTools.syntax.Antlr4Util$Grammar.visitDocument(Antlr4Util.scala:255)
        at wdlTools.syntax.v1_1.ParseTop.parseDocument(ParseTop.scala:1420)
        at wdlTools.syntax.v1_1.ParseAll.parseDocument(ParseAll.scala:399)
        ... 17 more

My WDL looks like this:

task fastQC {
    File fastqr1
    File fastqr2
    String outdir
    Int memory
    Int disk_space
    Int num_threads
    Int num_preempt
    String docker

    command {
        mkdir -p ${outdir}
        fastqc -o ${outdir} ${fastqr1}
        fastqc -o ${outdir} ${fastqr2}

        tar -cvzf ${outdir}.tar.gz ./${outdir}
    }
    output {
        File fastQC_report = '${outdir}.tar.gz'
    }
    runtime {
        docker: "${docker}"
        memory: "${memory}GB"
        disks: "local-disk ${disk_space} HDD"
        cpu: "${num_threads}"
        preemptible: "${num_preempt}"
    }
}

Replace custom URL class with java.net.URL

wdlTools.jar

Hello,

I am having trouble finding the location of the wdlTools.jar file. Could you please point me in the right direction?

Thank you!

format sub command errors on instantiation of structs

$ cat example.wdl
version 1.0

struct Message {
   String text
   Int priority
}

workflow my_workflow {
  input {
    String text
    Int priority
  }

  Message alert = {
    "text": text,
    "priority": priority
  }
}
$ wdltools check example.wdl 
$ wdltools format example.wdl 
[error] Command scallop failed
java.lang.Exception: Unrecognized type TypeIdentifier(Message)
        at wdlTools.generators.code.WdlFormatter$DataType$.fromWdlType(WdlFormatter.scala:825)
        at wdlTools.generators.code.WdlFormatter$DeclarationBase.<init>(WdlFormatter.scala:1231)
        at wdlTools.generators.code.WdlFormatter$DeclarationStatement.<init>(WdlFormatter.scala:1250)
        at wdlTools.generators.code.WdlFormatter$DeclarationStatement$.apply(WdlFormatter.scala:1249)
        at wdlTools.generators.code.WdlFormatter$DeclarationStatement$.apply(WdlFormatter.scala:1249)
        at scala.collection.immutable.Vector1.map(Vector.scala:1872)
        at scala.collection.immutable.Vector1.map(Vector.scala:375)
        at wdlTools.generators.code.WdlFormatter$DeclarationsSection.<init>(WdlFormatter.scala:1254)
        at wdlTools.generators.code.WdlFormatter.wdlTools$generators$code$WdlFormatter$$splitWorkflowElements(WdlFormatter.scala:1498)
        at wdlTools.generators.code.WdlFormatter$WorkflowSections.<init>(WdlFormatter.scala:1616)
        at wdlTools.generators.code.WdlFormatter$WorkflowBlock.body(WdlFormatter.scala:1656)
        at wdlTools.generators.code.WdlFormatter$BlockStatement.<init>(WdlFormatter.scala:1272)
        at wdlTools.generators.code.WdlFormatter$WorkflowBlock.<init>(WdlFormatter.scala:1652)
        at wdlTools.generators.code.WdlFormatter$DocumentSections.format(WdlFormatter.scala:1873)
        at wdlTools.generators.code.WdlFormatter.formatElement(WdlFormatter.scala:1898)
        at wdlTools.generators.code.WdlFormatter.formatDocument(WdlFormatter.scala:1907)
        at wdlTools.generators.code.WdlFormatter.$anonfun$formatDocuments$1(WdlFormatter.scala:1914)
        at wdlTools.syntax.WdlParser$Walker.addDocument$1(WdlParser.scala:66)
        at wdlTools.syntax.WdlParser$Walker.walk(WdlParser.scala:80)
        at wdlTools.generators.code.WdlFormatter.formatDocuments(WdlFormatter.scala:1913)
        at wdlTools.cli.Format.apply(Format.scala:15)
        at wdlTools.cli.Main$.runCommand(Main.scala:36)
        at wdlTools.cli.Main$.delayedEndpoint$wdlTools$cli$Main$1(Main.scala:48)
        at wdlTools.cli.Main$delayedInit$body.apply(Main.scala:12)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1(App.scala:76)
        at scala.App.$anonfun$main$1$adapted(App.scala:76)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:926)
        at scala.App.main(App.scala:76)
        at scala.App.main$(App.scala:74)
        at wdlTools.cli.Main$.main(Main.scala:12)
        at wdlTools.cli.Main.main(Main.scala)

$

Rewrite code formatter using concepts from dartfmt

http://journal.stuffwithstuff.com/2015/09/08/the-hardest-program-ive-ever-written/

Cannot resolve lint P004 "A top-level element is indented"

Hi. I'm trying to get a WDL 1.0 file to pass the wdlTools linter. I cannot figure out what changes I need to make in order to pass rule P004.

This small reproduction passes wdltools check ... and has been formatted by wdltools format ...:

version 1.0

workflow lint_test {
  input {
    String message
  }

  call my_task {
    input:
      message = message
  }
}

task my_task {
  input {
    String message
  }

  command <<<
	   printf 'The message is: %s\n' "~{message}"
  >>>

  output {
    String formatted = stdout()
  }

  runtime {
    docker: "debian:bullesye-slim"
  }
}

$ wdltools lint lint_test.wdl
==============================
Lint in lint_test.wdl
==============================
Line:Col | Rule | Description
------------------------------
1:11-2:3 | P004 | A top-level element is indented
12:1-13:3| P004 | A top-level element is indented
$ wdltools --version
wdlTools 0.17.7
$ java --version
openjdk 18 2022-03-22
OpenJDK Runtime Environment (build 18+36-2087)
OpenJDK 64-Bit Server VM (build 18+36-2087, mixed mode, sharing)
$

Any suggestions would be appreciated. Thanks!

How to call womtool in a Java web project.

Hello, I saw that you have packages stored in the Maven repository. Can I call the API in a Java web project?

Handle null value in typechecker

See discussion w/ Patrick here: openwdl/wdl#361

It is part of the 1.0 spec that null is a valid value in meta sections, but Patrick argues that this should be handled as a special case by tools. So the parser recognizes null as an identifier, but the typechecker should treat null as a valid identifier within the meta sections even if there is no declaration for an input/variable named 'null'.

API documentation

format sub-command removes some comments

I was expecting that the format sub-command would keep all comment lines (with whitespace added or removed to the left of the comment), but I'm seeing some comments get stripped out:

$ cat comment.wdl
version 1.0

workflow my_work {
  input{
    # first comment
    String foobar
  }

  Int num = 1
# second comment
}
$ wdltools check comment.wdl 
$ wdltools format comment.wdl 
comment.wdl
version 1.0

workflow my_work {
  input {
    # first comment
    String foobar
  }

  Int num = 1
}
$

Add "format" option for indentation level

Default indentation is 2 spaces. Any chance to add an option for number of spaces as I'd prefer 4?

Add support for (non-portable) UDFs

Add ability for user to define UDFs in Scala that are usable when the WDL is run via the task executor (including on DNAnexus)

Error when setting `Runtime.memory` as a string in a `development` version task

version development

task {
  ...
  runtime {
    memory: "1 GiB"
  }
}

throws exception:

wdlTools.eval.EvalException: Value V_String(1.0 GiB) could not be coerced to one of Vector(T_Int) at 164:12-164:31 in <string>
	at wdlTools.eval.Coercion$.$anonfun$coerceToFirst$1(Coercion.scala:209)
	at scala.Option.getOrElse(Option.scala:201)
	at wdlTools.eval.Coercion$.coerceToFirst(Coercion.scala:209)
	at wdlTools.eval.Eval.applyExprAndCoerce(Eval.scala:419)
	at wdlTools.eval.V2Runtime.getValue$1(Runtime.scala:275)
	at wdlTools.eval.V2Runtime.applyKv(Runtime.scala:291)
	at wdlTools.eval.Runtime.$anonfun$get$3(Runtime.scala:51)
	at scala.Option.orElse(Option.scala:477)
	at wdlTools.eval.Runtime.$anonfun$get$1(Runtime.scala:50)
	at scala.collection.immutable.Map$EmptyMap$.getOrElse(Map.scala:228)
	at wdlTools.eval.Runtime.get(Runtime.scala:47)
	at wdlTools.eval.V2Runtime.memory$lzycompute(Runtime.scala:344)
	at wdlTools.eval.V2Runtime.memory(Runtime.scala:343)
	at dx.core.languages.wdl.Runtime.$anonfun$parseInstanceType$3(Runtime.scala:104)
	at dx.core.languages.wdl.Runtime.$anonfun$parseInstanceType$3$adapted(Runtime.scala:104)
	at scala.Option.map(Option.scala:242)
	at dx.core.languages.wdl.Runtime.$anonfun$parseInstanceType$2(Runtime.scala:104)
	at scala.Option.getOrElse(Option.scala:201)
	at dx.core.languages.wdl.Runtime.parseInstanceType(Runtime.scala:103)
	at dx.executor.wdl.WdlTaskExecutor.getInstanceTypeRequest(WdlTaskExecutor.scala:182)
	at dx.executor.TaskExecutor.apply(TaskExecutor.scala:817)
	at dx.executor.BaseCli.dispatchCommand(BaseCli.scala:86)
	at dx.executor.BaseCli.main(BaseCli.scala:137)
	at dxExecutorWdl.MainApp$.delayedEndpoint$dxExecutorWdl$MainApp$1(Main.scala:27)
	at dxExecutorWdl.MainApp$delayedInit$body.apply(Main.scala:26)
	at scala.Function0.apply$mcV$sp(Function0.scala:39)
	at scala.Function0.apply$mcV$sp$(Function0.scala:39)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
	at scala.App.$anonfun$main$1(App.scala:76)
	at scala.App.$anonfun$main$1$adapted(App.scala:76)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:926)
	at scala.App.main(App.scala:76)
	at scala.App.main$(App.scala:74)
	at dxExecutorWdl.MainApp$.main(Main.scala:26)
	at dxExecutorWdl.MainApp.main(Main.scala)

Backport multi-line strings to WDL 1.2

Implement draft-2 code formater/generator

Inconsistent output from wdltools format

I'm trying to use wdltools format in continuous integration testing, in order to verify that the WDL file has been properly formatted. I've noticed that the output of

wdltools format test.wdl

generates a trailing newline, while

wdltools format --overwrite test.wdl

does not generate a file with a trailing newline. I would like to see the wdltools format output be independent of the --overwrite flag setting.

$ wdltools format --overwrite test.wdl
$ diff test.wdl <(wdltools format test.wdl)
0a1
> test.wdl
22c23
< }
\ No newline at end of file
---
> }
$ cat test.wdl
version 1.0

workflow my_workflow {
  input {
    String message
  }

  call display {
    input:
      message = message
  }
}

task display {
  input {
    String message
  }

  command <<<
    printf '~{message}'
  >>>
}$ wdltools format test.wdl
test.wdl
version 1.0

workflow my_workflow {
  input {
    String message
  }

  call display {
    input:
      message = message
  }
}

task display {
  input {
    String message
  }

  command <<<
    printf '~{message}'
  >>>
}
$

Add string literal type to ANTLR4 grammar and use it for import URI

Switch to RE2 for regular expression substitution in the sub() method

https://github.com/google/re2j

As per openwdl/wdl#243

Issues when cloning in a path with whitespaces

Hi,

I've run into troubles with installing / running wdlTools from a path with whitespaces.
First, ROOT_DIR was not set in the Makefile when trying to make on my MacOS system; It would set it to "".
I've replaced ROOT_DIR:=$(shell dirname $(realpath $(firstword $(MAKEFILE_LIST)))) with ROOT_DIR=$(shell pwd) and it worked.
Second, almost all tests failed with java.io.FileNotFoundException: Path does not exist

At this point, I decided to clone wdlTools elsewhere.

Cheers,
Laurent

Int x = i + i + i

Expected: ExprAdd(ExprAdd(ExprIdentifier(i),ExprIdentifier(i)), ExprIdentifier(i))

Actual: ExprAdd(ExprIdentifier(i),ExprIdentifier(i))

Possible fixes:

Generate nested structure as above
Make the value of operation types that may be chained a sequence rather than just a and b - this is nicer at least for formatting, because otherwise I need to detect consecutive uses of the same operation and merge them together

Struct literals

openwdl/wdl#297

Missing coercion of V_Directory

In wdlTools.eval.Coercion, around line 55, the coercion case (WdlTypes.T_Directory, d: V_Directory) => d is missing.

To reproduce, create a workflow that takes a struct like this as input:

struct Index {
    Directory? index_dir
    String? prefix
}