aclimatise / clihelpparser Goto Github PK

View Code? Open in Web Editor NEW

14.0 3.0 5.0 1.19 MB

Reads the output from CLI help commands, and generates machine readable schemas (CWL etc)

Home Page: https://aclimatise.github.io/CliHelpParser/

License: GNU General Public License v3.0

Python 100.00%

bioinformatics cwl wdl pipeline workflow parser cli

clihelpparser's People

Contributors

Stargazers

Watchers

Forkers

bernt-matthias moffmade blankenberg illusional leipzig

clihelpparser's Issues

Automatically add stdout as an output

This will require adding outputs to the data model

Test tool wrappers using cwltool/miniWDL

Maybe add test data as optional fields on the test case objects.

Check for an amount of valid dictionary words in the description

This should help filter out fake Positionals as is happening in #36.

missing dependencies

Was just trying to continue and needed to setup again. After this some modules were still missing:

pip install python-json-logger
pip install pygtail
pip install lark

still I get

AttributeError: module 'lark' has no attribute 'Tree'

Maybe more modules are missing in setup.py?

Add parser support for bwa-style default args

e.g. -t INT number of threads [1], has a default of 1

Provide detailed error objects

We have a few "expected" errors:

We killed the execution of the command because it timed out (in one of 3 ways)
We were given a string to parse that is too long

In these cases, we currently return an empty command which is implicitly "failed" since it has no arguments.

However, in these situations it is likely more helpful to throw an exception. This should be handled such that:

When running the CLI, this message is returned to the user (likely via logging)
When running via the Python API, this can be caught
When running in the Automation repo, we can store a structured error

Use the flag optional status correctly in CWL

e.g. an optional Flag is not converted to boolean?

Add `/` as a flag synonym delimiter

e.g. in bowtie2_build, -p/--packed

`taxsummary2kron` not terminating

bedtools misdetections

Tool:    bedtools random (aka randomBed)
Version: v2.26.0
Summary: Generate random intervals among a genome.

Usage:   bedtools random [OPTIONS] -g <genome>

Options: 
	-l	The length of the intervals to generate.
		- Default = 100.
		- (INTEGER)

	-n	The number of intervals to generate.
		- Default = 1,000,000.
		- (INTEGER)

these two options are currently detected as CliBoolean Flag.

Add reference to BaseCamp in aCLImatise docs

ideas for improving regular expression for determining parameter types

Currently key words min|max indicate an integer parameter, but this could be a float as well.

So one should check if these keyword are followed by an integer/float value (the latter maybe also in scientific notation). Or simply check if there is a float or integer on the line at all.

Choices are often represented (e.g. in argparse) by {A,B,C}, I often see ranges represented by [min,max], (min,max), [min:max], (min:max), where min and max ar integers / floats.

Allow overwriting model properties via json files

One might have a json file that stores curated information that helps aclimatise's parser, e.g.

{
    'parameter1': {
         'property1': 'value',
         'property2': 2
    },
    ...
}

would overwrite property1 and property2 in aclimatise's internal representation of parameter1. A simple example would be to overwrite help text, default values, file types.

With an appropriate 'encoding' also the parameter type might be changed:

{
    'parameter1': {
         '--TYPE--': 'CliFloat'
    },
    ...
}

could overwrite aclimatise's (wrong) guess of the parameter type (I suggest to use -- prefix since this should be an impossible name prefix).

In this way also hardcoding parameters, or removing parameters could be done via special parameter classes, e.g. CliHardcoded, CliNull (I try not to use the term blacklist here).

In the same way (with separate json files) also properties in the generated languages would be possible. Eg. a json file like the first exampe might overwrite/add attributes 'property1' and 'property2' of the xml tag with name="parameter1".

Avoid repeating subcommand testing

For example, typeHLA.js in the bwakit package:

INFO:acclimatise:Trying flags for typeHLA.js int
INFO:acclimatise:Trying typeHLA.js int --help
INFO:acclimatise:Trying typeHLA.js int -h
INFO:acclimatise:Trying typeHLA.js int
INFO:acclimatise:Trying typeHLA.js int --usage
INFO:acclimatise:The best help flag seems to be typeHLA.js int --help
INFO:acclimatise:typeHLA.js int does not seem to be a valid subcommand
INFO:acclimatise:Exploring typeHLA.js int
INFO:acclimatise:Trying flags for typeHLA.js int
INFO:acclimatise:Trying typeHLA.js int --help
INFO:acclimatise:Trying typeHLA.js int -h
INFO:acclimatise:Trying typeHLA.js int
INFO:acclimatise:Trying typeHLA.js int --usage
INFO:acclimatise:The best help flag seems to be typeHLA.js int --help
INFO:acclimatise:typeHLA.js int does not seem to be a valid subcommand
INFO:acclimatise:Exploring typeHLA.js int
INFO:acclimatise:Trying flags for typeHLA.js int
INFO:acclimatise:Trying typeHLA.js int --help
INFO:acclimatise:Trying typeHLA.js int -h
INFO:acclimatise:Trying typeHLA.js int
INFO:acclimatise:Trying typeHLA.js int --usage
INFO:acclimatise:The best help flag seems to be typeHLA.js int --help
INFO:acclimatise:typeHLA.js int does not seem to be a valid subcommand
INFO:acclimatise:Exploring typeHLA.js float
INFO:acclimatise:Trying flags for typeHLA.js float
INFO:acclimatise:Trying typeHLA.js float --help
INFO:acclimatise:Trying typeHLA.js float -h
INFO:acclimatise:Trying typeHLA.js float
INFO:acclimatise:Trying typeHLA.js float --usage
INFO:acclimatise:The best help flag seems to be typeHLA.js float --help
INFO:acclimatise:typeHLA.js float does not seem to be a valid subcommand
INFO:acclimatise:Exploring typeHLA.js string
INFO:acclimatise:Trying flags for typeHLA.js string
INFO:acclimatise:Trying typeHLA.js string --help
INFO:acclimatise:Trying typeHLA.js string -h
INFO:acclimatise:Trying typeHLA.js string
INFO:acclimatise:Trying typeHLA.js string --usage

Type inference of file types

e.g., look for arg.fasta which indicates that the CliFile is a fasta file.

This should become a new field on the CliFile class:

CliHelpParser/acclimatise/cli_types.py

Lines 86 to 91 in 435e560

    
           class CliFile(CliType): 
        
               """ 
        
               Takes a file path 
        
               """ 
        
               pass

Parsing should be implemented in <FlagArgSubclass>.get_type and/or infer_type:

CliHelpParser/acclimatise/model.py

Line 382 in 435e560

def infer_type(string) -> typing.Optional[cli_types.CliType]:

Don't use flag name if it's 2 characters

e.g. samtools coverage, the --rf flag should be called required_flags, not ff

Automatic inference of "input outputs"

e.g. --outfile out.fasta. This is both an input, and an output file. This should be added to the same part of the Command model as the output in #9.

Fix for name clashes, e.g. in `bedtools window`

Possible solution:

Generate names of minimum length
For each clashing name, compare descriptions using difflib, and remove the initial section of identical text, then rerun the initial generation

Certain containers never terminate

Problematic containers include:

quay.io/biocontainers/apt-probeset-summarize:2.10.0--0
quay.io/biocontainers/gmap:2020.06.01--pl526h2f06484_1
quay.io/biocontainers/dcmtk:3.6.1--hc6cd04f_6
quay.io/biocontainers/codonw:1.4.4--h470a237_1
quay.io/biocontainers/bwakit:0.7.17.dev1--0
quay.io/biocontainers/discovardenovo:52488--1

Add an actual timeout to the docker executor

Currently it only times out if no input is received, but in the case of the yes command, it will never terminate.

Support `man` pages

Currently we only run commands directly with help flags. Support checking for man pages.

Add to bio.tools

Add aCLImatise as a tool in the bio.tools registry.

Support docker images

Images will have to be added as a field to the data model, and then added to the CWL and WDL converters

Fix for infinite loop in certain tools, like `dinosaur` and `mauve`

Rework the versions to prevent dependency issues

Fix the CWL ids

Fix the filepath .. from each of the CWL id fields

'NoneType' object has no attribute 'decode'

        Exploring guppy...Done.                                                                                                                                                                                    
                Acclimatising the command guppy failed with error Traceback (most recent call last):
                  File "/home/ubuntu/BiocondaCli/bioconda_cli/util.py", line 134, in acclimatise_exe
                    cmd = explore_command(cmd=[exe], executor=exec)
                  File "/home/ubuntu/BiocondaCli/venv/lib/python3.7/site-packages/acclimatise/__init__.py", line 206, in explore_command
                    try_subcommand_flags=try_subcommand_flags,
                  File "/home/ubuntu/BiocondaCli/venv/lib/python3.7/site-packages/acclimatise/__init__.py", line 182, in explore_command
                    command = best_cmd(cmd, flags, executor=executor)
                  File "/home/ubuntu/BiocondaCli/venv/lib/python3.7/site-packages/acclimatise/__init__.py", line 104, in best_cmd
                    final = executor.execute(help_cmd)
                  File "/home/ubuntu/BiocondaCli/venv/lib/python3.7/site-packages/acclimatise/execution/docker.py", line 40, in execute
                    return (stdout or stderr).decode()
                AttributeError: 'NoneType' object has no attribute 'decode'

Add Galaxy output format support

Add to biocontainers

PR some code into biocontainers that will pull the latest YAML for each executable and list the parameters.

Change the PyPI package, and module name to `aclimatise` instead of `acclimatise`

It's confusing.

AttributeError: 'str' object has no attribute 'repeatable'

                  File "/mnt/miniconda3/lib/python3.7/site-packages/acclimatise/usage_parser/__init__.py", line 20, in parse_usage                                                                        [50/1813]
                    toks = usage.setDebug(debug).searchString(text)                                                                                                                                                
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 2103, in searchString                                                                                                      
                    return ParseResults([t for t, s, e in self.scanString(instring, maxMatches)])                                                                                                                  
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 2103, in <listcomp>                                                                                                        
                    return ParseResults([t for t, s, e in self.scanString(instring, maxMatches)])                                                                                                                  
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 2007, in scanString                                                                                                        
                    nextLoc, tokens = parseFn(instring, preloc, callPreParse=False)                                                                                                                                
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 1683, in _parseNoCache                                                                                                     
                    loc, tokens = self.parseImpl(instring, preloc, doActions)                                                                                                                                      
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 4069, in parseImpl                                                                                                         
                    loc, exprtokens = e._parse(instring, loc, doActions)                                                                                                                                           
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 1683, in _parseNoCache                                                                                                     
                    loc, tokens = self.parseImpl(instring, preloc, doActions)                                                                                                                                      
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 4707, in parseImpl                                                                                                         
                    loc, tmptokens = self_expr_parse(instring, preloc, doActions)                                                                                                                                  
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 1683, in _parseNoCache                                                                                                     
                    loc, tokens = self.parseImpl(instring, preloc, doActions)                                                                                                                                      
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 4462, in parseImpl                                                                                                         
                    return self.expr._parse(instring, loc, doActions, callPreParse=False)                                                                                                                          
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 1683, in _parseNoCache
                    loc, tokens = self.parseImpl(instring, preloc, doActions)
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 4164, in parseImpl
                    loc2, toks = expr1._parse(instring, loc, doActions)
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 1683, in _parseNoCache
                    loc, tokens = self.parseImpl(instring, preloc, doActions)
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 4069, in parseImpl
                    loc, exprtokens = e._parse(instring, loc, doActions)
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 1683, in _parseNoCache
                    loc, tokens = self.parseImpl(instring, preloc, doActions)
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 4697, in parseImpl
                    loc, tokens = self_expr_parse(instring, loc, doActions, callPreParse=False)
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 1683, in _parseNoCache
                    loc, tokens = self.parseImpl(instring, preloc, doActions)
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 4462, in parseImpl
                    return self.expr._parse(instring, loc, doActions, callPreParse=False)
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 1683, in _parseNoCache
                    loc, tokens = self.parseImpl(instring, preloc, doActions)
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 4164, in parseImpl
                    loc2, toks = expr1._parse(instring, loc, doActions)
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 1716, in _parseNoCache
                    tokens = fn(instring, tokensStart, retTokens)
                  File "/mnt/miniconda3/lib/python3.7/site-packages/pyparsing.py", line 1316, in wrapper
                    ret = func(*args[limit[0]:])
                  File "/mnt/miniconda3/lib/python3.7/site-packages/acclimatise/usage_parser/elements.py", line 132, in visit_list_element
                    el.repeatable = True
                AttributeError: 'str' object has no attribute 'repeatable'

Better logic for combining types from description, argument and names

For instance, for the following parameter from samtools sort:

-o FILE    Write final output to FILE rather than standard output

only "FILE" is sent to the regular expressions. But in order to make #15 work also the description would be needed.

$ samtools.pl showALEN --usage
Can't open --usage: No such file or directory at /media/michael/Storage2/miniconda/envs/aclimatise-test/bin/samtools.pl line 50.