common-workflow-lab / galaxy Goto Github PK

Extend Galaxy XML syntax as follows:

<container type="docker" load="import_url">imageID</container>

doc: "Specify a HTTP URL from which to download a Docker image using docker load."

CWLtool implementation

        elif "dockerLoad" in dockerRequirement:
            cmd = ["docker", "load"]
            _logger.info(str(cmd))
            if not dry_run:
                if os.path.exists(dockerRequirement["dockerLoad"]):
                    _logger.info("Loading docker image from %s", dockerRequirement["dockerLoad"])
                    with open(dockerRequirement["dockerLoad"], "rb") as f:
                        loadproc = subprocess.Popen(cmd, stdin=f, stdout=sys.stderr)
                else:
                    loadproc = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=sys.stderr)
                    _logger.info("Sending GET request to %s", dockerRequirement["dockerLoad"])
                    req = requests.get(dockerRequirement["dockerLoad"], stream=True)
                    n = 0
                    for chunk in req.iter_content(1024*1024):
                        n += len(chunk)
                        _logger.info("\r%i bytes" % (n))
                        loadproc.stdin.write(chunk)
                    loadproc.stdin.close()
                rcode = loadproc.wait()
                if rcode != 0:
                    raise process.WorkflowException("Docker load returned non-zero exit status %i" % (rcode))
                found = True

WIP: in this branch - see 0daf498.

Demonstrates using the job description to populate environment variables.

Parsing and stuff is there and test case written (test_cwl_env_tool1) but implementation remains a little tricky.

Galaxy queries on imageId as the result of dockerPull.

This is what CWLTool defaults to as well, but Galaxy should be updated to allow a different image ID then pull target.

if "dockerImageId" not in dockerRequirement and "dockerPull" in dockerRequirement:
        dockerRequirement["dockerImageId"] = dockerRequirement["dockerPull"]

How CWLTool uses imageId for search.

for ln in subprocess.check_output(["docker", "images", "--no-trunc", "--all"]).splitlines():
        try:
            m = re.match(r"^([^ ]+)\s+([^ ]+)\s+([^ ]+)", ln)
            sp = dockerRequirement["dockerImageId"].split(":")
            if len(sp) == 1:
                sp.append("latest")
            # check for repository:tag match or image id match
            if ((sp[0] == m.group(1) and sp[1] == m.group(2)) or dockerRequirement["dockerImageId"] == m.group(3)):
                found = True
                break
        except ValueError:
            pass

So the extension in Galaxy land is likely to keep the XML body as the imageID and optionally allow overriding the pull location:

<container type="docker" pull="dockerPull">imageID</container>

Extend Galaxy XML syntax as follows:

<container type="docker" import="import_url">imageID</container>

doc: "Provide HTTP URL to download and gunzip a Docker images using `docker import."

CWLtool implementation

        elif "dockerImport" in dockerRequirement:
            cmd = ["docker", "import", dockerRequirement["dockerImport"], dockerRequirement["dockerImageId"]]
            _logger.info(str(cmd))
            if not dry_run:
                subprocess.check_call(cmd, stdout=sys.stderr)
                found = True

The changes in the run up to 1.0 completely broke how I was using generatefiles and such. Started work here - https://github.com/common-workflow-language/galaxy/tree/cwl-1.0.

CWL tools can take over 2 seconds to load on a local SSD disk. Comparable Galaxy tools load in milliseconds.

galaxy.tools.parser.factory INFO 2016-02-04 09:43:39,420 Loading CWL tool - this is experimental - tool likely will not function in future at least in same way.
galaxy.tools.parser.cwl WARNING 2016-02-04 09:43:41,689 TOOL ID is cat1-tool
galaxy.tools.parser.factory INFO 2016-02-04 09:43:41,690 Loading CWL tool - this is experimental - tool likely will not function in future at least in same way.
galaxy.tools.parser.cwl WARNING 2016-02-04 09:43:44,106 TOOL ID is cat2-tool
galaxy.tools.parser.factory INFO 2016-02-04 09:43:44,107 Loading CWL tool - this is experimental - tool likely will not function in future at least in same way.
galaxy.tools.parser.cwl WARNING 2016-02-04 09:43:46,412 TOOL ID is cat3-tool
galaxy.tools.parser.factory INFO 2016-02-04 09:43:46,413 Loading CWL tool - this is experimental - tool likely will not function in future at least in same way.
galaxy.tools.parser.cwl WARNING 2016-02-04 09:43:48,709 TOOL ID is cat4-tool
galaxy.tools.parser.factory INFO 2016-02-04 09:43:48,710 Loading CWL tool - this is experimental - tool likely will not function in future at least in same way.
galaxy.tools.parser.cwl WARNING 2016-02-04 09:43:51,018 TOOL ID is bwa-mem-tool
galaxy.tools.parser.factory INFO 2016-02-04 09:43:51,019 Loading CWL tool - this is experimental - tool likely will not function in future at least in same way.
galaxy.tools.parser.cwl WARNING 2016-02-04 09:43:53,478 TOOL ID is env-tool1
galaxy.tools.parser.factory INFO 2016-02-04 09:43:53,478 Loading CWL tool - this is experimental - tool likely will not function in future at least in same way.
galaxy.tools.parser.cwl WARNING 2016-02-04 09:43:55,793 TOOL ID is env-tool2
galaxy.tools.parser.factory INFO 2016-02-04 09:43:55,794 Loading CWL tool - this is experimental - tool likely will not function in future at least in same way.

Need to start on workflow conversion.

• Implement a conformance test for this.
• Ensure the conformance test passes.

xref #7

cwl-runner tool.cwl job.json

Initial and entirely terrible first crack at this is here abbae34. It will be rebased into the main CWL commit.

• Add a conformance test.
• Implement and pass the conformance test.
• Upstream Conformance test.

@jmchilton TODO: describe how to do this.

Like a simpler version of this tool:

https://github.com/common-workflow-language/common-workflow-language/blob/master/draft-3/draft-3/optional-output.cwl

Collect Output:

• Ensure relocate iterates over secondary output files.
  {'path': '/home/john/workspace/galaxy/database/job_working_directory/000/42/input.txt', 'size': 1, 'secondaryFiles': [{'path': '/home/john/workspace/galaxy/database/job_working_directory/000/42/input.txt.idx1', 'class': 'File'}], 'class': 'File', 'checksum': 'sha1$b6589fc6ab0dc82cf12099d1c2d40ab994e8410c'}
• Persist dataset name to output map, CWL doesn't know about the ID only the name.
• Move each secondaryFile <f> to <job_working_directory>/dataset_<id>_files/__secondaryFiles__/f.

Ensure Input:

• Check for secondary files when preparing the job.
• Link in if any secondary files are referenced (where, how can I tell, how can I link them).

Use parseInt.cwl as an example.

xref #27 (Galaxy native version of this)
xref #5 (Has checklist item for parseInt)

This was the first attempt at multi-input scattering - ae11f56 - now that I'm staring at count-lines 6 and 7 it is clear to me that replacing CollectionInfo with a new type of lets say tracker of scattering and input collections is wrong. We should use CollectionInfo and all of its utilities for tracking - we should just feed it "ephemeral" collections created just for tool execution. This has some distinct advantages over the first approach:

• We can track these collections in the DB at some point for tracking the structure of what we did.
• We can build arbitrary ephemeral collections in response to scatter input expressions. e.g. scatter-valuefrom-wf1.
• We can handle non-scattering operations with Ephemeral collections as well (such as supplying multiple lists to a single list input in a merged fashion) - e.g. count-lines7.

Galaxy tools should allow building from a specified Dockerfile.

        elif "dockerFile" in dockerRequirement:
            dockerfile_dir = tempfile.mkdtemp()
            with open(os.path.join(dockerfile_dir, "Dockerfile"), "w") as df:
                df.write(dockerRequirement["dockerFile"])
            cmd = ["docker", "build", "--tag=%s" % dockerRequirement["dockerImageId"], dockerfile_dir]
            _logger.info(str(cmd))
            if not dry_run:
                subprocess.check_call(cmd, stdout=sys.stderr)
                found = True

Extend Galaxy XML syntax as follows:

<container type="docker" dockerfile="Dockerfile">imageID</container>

When CWL tools are clicked on in the tool panel a weird window-in-window thing happens. Only happen for CwlTools not for normal Tools. tool.tool_type is different.

Already made one fix to prevent tool from just submitting. 1315239

Grepping around I can't find any place where the behavior is dispatching on tool_type, any clues @guerler

Needed by bcbio workflow and other GA4GH workflows.

http://www.commonwl.org/v1.0/Workflow.html#StepInputExpressionRequirement

• step-valuefrom-wf.cwl ./run_tests.sh -api test/api/test_cwl_conformance_v1_0.py:CwlConformanceTestCase.test_conformance_v1_0_61
• step-valuefrom2-wf.cwl ./run_tests.sh -api test/api/test_cwl_conformance_v1_0.py:CwlConformanceTestCase.test_conformance_v1_0_62
• step-valuefrom3-wf.cwl ./run_tests.sh -api test/api/test_cwl_conformance_v1_0.py:CwlConformanceTestCase.test_conformance_v1_0_63

Will create a follow up issue on implementing over scattering - it looks like at least the BCBIO workflows do not require scattering and valueFrom on the same step. The vision chipseq workflows do require this however.

Review CWL reference and cwltool as well as support in this branch for Docker, describe the limitations in terms of specific commands to run and create an issue. @jmchilton can probably fix this quickly if the limitations are laid out clearly.

No need to modify the tool panel directly, just use GALAXY_CONFIG_ to load samples.

First cut of this just for CLI usage could be to upgrade StepParameters to allow this. Need to consider how to build a GUI for these down the road.

Most basic workflow in the test CWL test suite.

Depends on #42
xref #5

There really isn't a comparable thing in Galaxy's workflow model. Need to track:

• default
• valueFrom
• linkMerge (defaults to merge_nested but can be merge_flattened) - countlines 6 and 7.

• Ensure Galaxy options for expression evaluation are respected in runtime actions for expression tools.
• Ensure problems encountered during evaluation of expression tools are handled well, job should fail with a reasonable message. Write a test case.

• Refactor Galaxy's workflow import stuff to abstract representation.
  • Merge build_workflow_from_dict and update_workflow_from_dict for creating model.Workflow(). (galaxyproject#1735)
  • Allow creation from a file (.ga at first).
• Allow toolbox to create workflows from files.
• Track transient file path and expose in the API.

Don't load unsupported tools.

WIP in this branch 29cff11.

Extend CWL tool support for this new Galaxy feature.

Galaxy sticks a bunch of stuff in the job working directory that should not be there for CWL jobs.

Obviously file paths need to be replaced with something Galaxish - but there is so much logic in converting the CWL job representation to a Galaxy tool request that it should be handled by Galaxy itself (since it has the tool representation).

• binding-test.cwl
• bwa-mem-job.json
• bwa-mem-tool.cwl
• cat-job.json
• cat-n-job.json (1e2385b)
• cat1-tool.cwl
• cat2-tool.cwl (3be2d69)
• cat3-tool.cwl
• cat4-tool.cwl (a255442)
• cat5-tool.cwl
• count-lines1-wf.cwl
• count-lines2-wf.cwl
• count-lines3-job.json
• count-lines3-wf.cwl
• count-lines4-job.json
• count-lines4-wf.cwl
• count-lines5-wf.cwl
• count-lines6-job.json
• count-lines6-wf.cwl
• count-lines7-wf.cwl
• count-lines8-wf.cwl
• count-lines9-wf.cwl
• echo-tool.cwl
• empty.json
• env-job.json (e5c5f9b)
• env-tool1.cwl
• env-tool2.cwl (e5c5f9b)
• env-wf1.cwl
• env-wf2.cwl
• foaf.rdf
• formattest-job.json
• formattest.cwl
• formattest2-job.json
• formattest2.cwl
• formattest3.cwl
• hello.txt
• index.py
• metadata.cwl
• number.txt
• optional-output.cwl
• params.cwl
• params2.cwl
• params_inc.yml
• parseInt-job.json (f4b5e8e)
• parseInt-tool.cwl (f4b5e8e)
• rename-job.json
• record-output-job.json
• record-output.cwl
• ref.fasta
• rename-job.json (ac74d46)
• rename.cwl (ac74d46)
• revsort-job.json
• revsort.cwl
• revtool.cwl
• scatter-job1.json
• scatter-job2.json
• scatter-wf1.cwl
• scatter-wf2.cwl
• scatter-wf3.cwl
• scatter-wf4.cwl
• search-job.json
• search.cwl
• search.py
• shelltest.cwl
• sorttool.cwl
• template-tool.cwl
• tmap-job.json
• tmap-tool.cwl
• wc-job.json (0a89778)
• wc-tool.cwl (0a89778)
• wc2-tool.cwl (6a0460e)
• wc3-tool.cwl (692c46f)
• wc4-tool.cwl (692c46f)

Extend Galaxy XML syntax as follows:

<container type="docker" output_directory="directory">imageID</container>

CWL definition of attribute:

    - name: dockerOutputDirectory
      type: ["null", "string"]
      doc: |
        Set the designated output directory to a specific location inside the
        Docker container.

• Document what it should look like and how to access using cwltool data structures.

Draft 3 will not have generic expressions, it will only be JavaScript. So while I would still like to target "draft 2" here, I'm just going to assume all expressions are JavaScript expressions.

Like the generic case of this:

https://github.com/common-workflow-language/common-workflow-language/blob/master/draft-3/draft-3/optional-output.cwl

["null", File] should probably be handled specially.

Demonstrates handling very simple ["null", "<simple_type>"] type unions using conditionals.

Implemented and rebased into 1e41f8b.

Required for biocadie workflow for GA4GH.

Galaxy needs to support CWL directory types.

Required for Biocadie workflow to GA4GH.

Hey @jmchilton

Lets schedule a time to work on this. Send me a Google Calendar invite and you should be able to see my availability.

Running issue of junk I want to refactor.

• Move all CWL tool tests to their own file, update commit message.
• Relabel OutputInstance fields.