crossmap with multiple threads about metagem HOT 2 CLOSED

Hi Francisco,

Thanks a lot for the rapid response. Actually, I am running metaGEM with --local flag so I only made changes in the config.yaml file. But maybe I will give it a try by running it in parallel and also have a look the new tool.

Thanks a lot.
Kunal

from metagem.

Hi Kunal,

There are sections of code in the crossmap rule that probably do not make full use of parallelisation, e.g. when combining multiple sorted bam files with jgi_summarize_bam_contig_depths or concoct_coverage_table.py, although there is not much I can do about this. I would suggest that you monitor your job with htop to see if at least the bwa-mem commands are getting multiples cores. Also, it is possible that you may have properly assigned your cores in the config.yaml file (used for workflow parameters) but NOT in the cluster_config.json file, which is the one that actually requests resources from your cluster. Please double check these files.
Also how large is your dataset? For e.g. more than 250 samples it becomes difficult to obtain cross mapping results in a reasonable amount of time. By default metaGEM runs crossmapping in series (i.e. one job per focal sample, then each job maps short read samples to the focal sample assembly one at a time), have you considered the cross mapping in parallel? However, again for large datasets this will spawn a huge number of jobs that could take a while to finish on the cluster. If I were you, I would use the default crossmapping in series approach, but instead of bwa-mem I would use this new fairy tool that just came out that is apparently ~250x faster. Seems easy to use and is compatible with binning tools, here is the preprint and github repo if you are interested. I will definitely replace bwa-mem with this tool in the metaGEM workflow once I have some time in the future, but in the meantime please have a look.

Best,
Francisco

from metagem.