shuang-broad / canu-wdl Goto Github PK
View Code? Open in Web Editor NEWThis project forked from marbl/canu
A customized fork of canu 1.9 for WDL-ization in the cloud
Home Page: http://canu.readthedocs.io/
This project forked from marbl/canu
A customized fork of canu 1.9 for WDL-ization in the cloud
Home Page: http://canu.readthedocs.io/
canu::Configure::configureAssembler()
canu::Defaults::setDefaults()
canu::Defaults::setExeDefaults()
per this discussion
Not so easy task, but reduces maintenance headache.
The repartition step is I/O bound, takes approximately 2 hours, using no more than 4 threads and not much memory.
We may lower cost in this step by:
meryl-count
Perfect candidate for data-parallelization. The majority of observed ~ 30 minutes/batch is actually spent on loading the parental reads into memory with 1 thread. And 30 GB memory has been observed to have been enough per batch. Reading the perl code reveals that the memory requirement is purely dependent on the number of files per batch (the choice of
We may lower cost in this step by a two-step break up of the current implementation
Step one
exit gracefully immediately after
meryl-count.sh
, meryl-count.memory
meryl-merge.sh
& meryl-subtract.sh
are generated.This requires us to implement something similar to stopAfter("trio-binning scripts are configured")
.
Step two
a following scatter call to meryl-count tasks (in the batch-count number of ways), each VM taking in a batch-definition file, and localize just that. Launch meryl-count.sh
on the given batch with the correct batch id, then yield. Remember to save the count files and the log ("out") files in each batch.
meryl-merge
& meryl-subtract
This is essentially a gather step after the scatter in meryl-count
above. The task essentially just need to take the outputs of meryl-count
above, and perform the merge and subtract task.
Not exactly heavy computations based on test run on an VM. The test run tells us that 30 GB is OK for the task to finish, in 2~3 hours.
Seemingly can be finished in 1 hour on a rich machine. Test run with 32 threads and 256 GB memory finished in ~ 1 hour.
Right now the two parents short reads are re-partitioned in serial.
We can speed up by parallelize that, one task per parent.
Expecting a drop of ~ 1 hour runtime.
But lower priority.
The following subroutines or blocks should be understood first before meaningful changes to the code can happen.
Relevant blocks:
while (scalar(@ARGV))
: parses cmd line options; see comment block for canu::Defaults::addSequenceFile
for relevant readingif ((scalar(@haplotypes) > 0) && (setOptions($mode, "haplotype") eq "haplotype"))
: the short block that actually calls out to canu/HaplotypeReads
for trio binning.addSequenceFile
: parsing path of parental and child reads, DOES NOT actually load readsgetPhysicalMemorySize
& getNumberOfCPUs
: literally detects the number of CPUs and memory available on the machine; they are only non-trivially used in Configure::getAllowedResources()
setDefaults
: see doc for Defaults.pm
belowsetExecDefaults($$)
: sets Memory, Threads, StageSpace and Concurrency of: 1) a specific stage with 2) description to undef
(yes, all four).checkParameters
: makes sure options and parameters make senseconfigureAssembler
: see docs for Configure.pm
belowgetAllowedResources
: see docs for Configure.pm
belowfindGridMaxMemoryAndThreads
and expandRange
: see docs for Configure.pm
belowsubmitScript
: returns without doing anything if no grid is detected in canu::Defaults
, which is our intended use case.getBinDirectoryShellCode
, setWorkDirectoryShellCode
, & getJobIDShellCode
stopAfter
: checks that given "stop after" stage is requested by user, and exit the program.submitOrRunParallelJob
: see docs for Execution.pm
belowhaplotypeReadsExist
haplotypeSplitReads
haplotypeCountConfigure
haplotypeCountCheck
haplotypeMergeCheck
haplotypeSubtractCheck
haplotypeReadsConfigure
haplotypeReadsCheck
Logical steps in canu.pl
options parsing, parameters setting, resources detection
minio IO prep
reads haplotyping
otherwise it continues silently, hurts debugging.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.