microbiomedata / mg_annotation Goto Github PK
View Code? Open in Web Editor NEWMetagenome Annotation Workflow
Metagenome Annotation Workflow
JGI is on annotation version 5.1.*, NMDC is still on version 5.0. Update workflow to be up to date with JGI.
Deliverable this task is associated with
1
RACI
Responsible: @Michal-Babins
Accountable: @aclum
Consulted: @hubin-keio
Informed:
Describe the task?
ko_ec_img_nr_db
ko_ec_md5_mapping
ko_ec_taxon_to_phylo_mapping
Criteria for completion
A test annotation workflow run completes successfully and the imgap_version file has a img-nr database version has a year matching 2023
Completion Date (Goal)
July 7
Target Sprint Start & End Dates
Start: July 3
End: July 14
Tag Blocker/Contingent upon issues
Dependent on new database version release from @mhuntemann
Add github actions to create releases for mg annotation.
We've decided to move away from specifying runtime attributes in an options.json, instead defaults should be specified in the wdl at the task level. JAWS does not and will not support specifying a cromwell options.json because they want flexibility to change the workflow runner in the future.
create a large test data set with expected outcome if workflow run is successful
link test results to this issue
Example inputs.json in both the master and werkflow branch is not valid.
The first issue is that there is an extra closing bracket so the file is not valid json.
The second issue is that 'annotation.imgap_input_fasta' is not valid for the annotation.wdl, it should be 'annotation.input_file' based on the wdl or vice versa. Either the inputs.json or the wdl needs to be updated.
Update WDL's to use Version 1.0
Marcel next week should be providing an updated version that outputs a lineage file. The wdl workflow should be updated to have this as an output and the runtime automation should make a data object for this file. A file enum,Lineage sdb, has been made already.
Blocked on updates from Marcel and runtime API being updated to schema version 7.7
create a small test data set with expected outcome if workflow run is successful
link test results to this issue
There are a few things we need to confirm are working properly so we can support NMDC workflows that start with an input assembly file. That is, workflows that start with a contigs file, instead of reads.
In this case we need to
see related task on the automation side microbiomedata/nmdc_automation#35
Add github action to run miniwdl to lint main wdl file.
The current docker container pulls a version of CRT from a nersc ftp site, Marcel has made this repo public so we should pull and build the code from the public repo. CRT version CRT-CLI_v1.8.4
repo: https://code.jgi.doe.gov/img/img-pipelines/crt-cli-imgap-version
The merge of wdl 1.0 duplicated some output values in crt.wdl of the run task.
see minwdl linting
miniwdl check ./annotation_full.wdl
(./annotation_full.wdl Ln 3 Col 1) Failed to import ./structural-annotation.wdl
(./structural-annotation.wdl Ln 5 Col 1) Failed to import crt.wdl
(crt.wdl Ln 48 Col 5) Multiple declarations of crisprs
File crisprs = "{prefix}_crt.crisprs"{prefix}_crt.gff"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(crt.wdl Ln 49 Col 5) Multiple declarations of gff
File gff = "
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(crt.wdl Ln 50 Col 5) Multiple declarations of crt_out
File crt_out = "~{prefix}_crt.out"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Make the following updates to the dockerfile, build new docker file, update wdls to use new docker image
update to Marcel's patched version of prodigal
update to CRT-CLI_v1.8.4
update to pulling from public repo instead of nersc portal for IMG wrapper scripts.
We missed updating to the new product mapping tables when we updated to img annotation version 5.2.
Update the workflow to use /global/cfs/cdirs/m3408/refdata/img/Product_Name_Mappings/20230814
This is what Marcel is now using.
cc @hubin-keio
The edge-nmdc website needs to display the output files for the annotation workflow. Currently, the output files for the mg-annotation workflow remain within the cromwell-execution directory after each task is completed. A task is needed to copy files from the cromwell-execution directory to a directory in edge_app/nmdc-edge/projects
. This includes adding outdir
as an optional input to the wdl and inputs.json files
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.