Dear Francisco, I realize metaGEM was optimized to run on a HPC clus

Hey Sam, Good question. If you look at the <code class="notranslate"

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

How to run on a cluster without workload manager about metagem HOT 4 CLOSED

franciscozorrilla commented on September 3, 2024

How to run on a cluster without workload manager

from metagem.

Comments (4)

franciscozorrilla commented on September 3, 2024

Hey Sam,

Good question. If you look at the metaGEM.sh script, specifically in line 358, you will see that only 1 job will be submitted at a time when running metaGEM with the --local flag:

metaGEM/metaGEM.sh

Lines 338 to 364 in a125bb7

    
           # Submit local function, similar to submitLogin() but can handle wildcard expanded rules for non-cluster usage 
        
           submitLocal() { 
        
               # Parse Snakefile rule all (line 22 of Snakefile) input to match output of desired target rule stored in "$string". Note: Hardcoded line number. 
        
               echo "Parsing Snakefile to target rule: $task ... " 
        
               sed  -i "22s~^.*$~        $string~" Snakefile 
        
               checkParams 
        
               snakeConfig 
        
               echo "Unlocking snakemake ... " 
        
               snakemake --unlock -j 1 
        
               echo -e "\nDry-running snakemake jobs ... " 
        
               snakemake all -n 
        
               while true; do 
        
                   read -p "Do you wish to submit this batch of jobs on your local machine? (y/n)" yn 
        
                   case $yn in 
        
                       [Yy]* ) echo "snakemake all -j 1 -k"|bash; break;; 
        
                       [Nn]* ) exit;; 
        
                       * ) echo "Please answer yes or no.";; 
        
                   esac 
        
               done 
        
           }

Let me know if you run into any issues with this.

Best,
Francisco

from metagem.

slambrechts commented on September 3, 2024

Hi @franciscozorrilla,

Thank you for your answer. Ok, so it should make sense then to set the number of cores for all jobs to the maximum number of threads I am allowed to use at any time (e.g. 20)? Otherwise, if I use the default values, when fastp is running for example, only 4 cores will be used, and 16 will be left unused?

Best,
Sam

from metagem.

franciscozorrilla commented on September 3, 2024

Hi Sam,

That is correct. When running metaGEM with the --local flag on a workstation (or a cluster without a scheduler), you should set the max number of threads you are allowed to use for each task in the config.yaml file. The metaGEM.sh parser only configures the number of cores in the cluster_config.json file, so you need to manually modify the number of threads in these lines of the config.yaml file:

metaGEM/config.yaml

Lines 42 to 59 in d81186a

    
           cores: 
        
               fastp: 4 
        
               megahit: 48 
        
               crossMap: 48 
        
               concoct: 48 
        
               metabat: 48 
        
               maxbin: 48 
        
               refine: 48 
        
               reassemble: 48 
        
               classify: 2 
        
               gtdbtk: 48 
        
               abundance: 16 
        
               carveme: 4 
        
               smetana: 12 
        
               memote: 4 
        
               grid: 24 
        
               prokka: 2 
        
               roary: 12

Hope that is clear and let me know if you run into any issues with this!

Best,
Francisco

from metagem.

slambrechts commented on September 3, 2024

Hi Francisco,

Yes, very clear, thanks! When running fastp however, it seems 16 threads are used instead of 20. I get a warning that I specified 20 but only 16 will be used. But this seems to be inherent to fastp.

Best,
Sam

from metagem.

Recommend Projects

How to run on a cluster without workload manager about metagem HOT 4 CLOSED

Comments (4)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	# Submit local function, similar to submitLogin() but can handle wildcard expanded rules for non-cluster usage
	submitLocal() {

	# Parse Snakefile rule all (line 22 of Snakefile) input to match output of desired target rule stored in "$string". Note: Hardcoded line number.
	echo "Parsing Snakefile to target rule: $task ... "
	sed -i "22s~^.*$~ $string~" Snakefile

	checkParams

	snakeConfig

	echo "Unlocking snakemake ... "
	snakemake --unlock -j 1

	echo -e "\nDry-running snakemake jobs ... "
	snakemake all -n

	while true; do
	read -p "Do you wish to submit this batch of jobs on your local machine? (y/n)" yn
	case $yn in
	[Yy]* ) echo "snakemake all -j 1 -k"\|bash; break;;
	[Nn]* ) exit;;
	* ) echo "Please answer yes or no.";;
	esac
	done

	}

	cores:
	fastp: 4
	megahit: 48
	crossMap: 48
	concoct: 48
	metabat: 48
	maxbin: 48
	refine: 48
	reassemble: 48
	classify: 2
	gtdbtk: 48
	abundance: 16
	carveme: 4
	smetana: 12
	memote: 4
	grid: 24
	prokka: 2
	roary: 12