Improving modularisation by allowing inputs from the main workflow into subworkflows about snakemake HOT 4 CLOSED

It is a fragile hack, but rather than importing submodules you can just make a rule in your top snakefile that runs the submodule directly. This would also solve your DAG visualiation, because the complex submodules only appear as the single rule “Create_transport_timeseries” etc (which then spawn all their own jobs when run). include: <submodule> can sometimes slow DAG building too much or have naming clashes, so each approach has its own drawbacks.

rule Create_transport_timeseries:
    input:
        <all input needed to start submodule>
    output:
       <any output you want from the entire submodule>
    shell:
        'snakemake -s <submodule> ...'

from snakemake.

It is a fragile hack, but rather than importing submodules you can just make a rule in your top snakefile that runs the submodule directly. This would also solve your DAG visualiation, because the complex submodules only appear as the single rule “Create_transport_timeseries” etc (which then spawn all their own jobs when run). include: <submodule> can sometimes slow DAG building too much or have naming clashes, so each approach has its own drawbacks.

rule Create_transport_timeseries:
    input:
        <all input needed to start submodule>
    output:
       <any output you want from the entire submodule>
    shell:
        'snakemake -s <submodule> ...'

Thank you for the input. This might work! Testing shows some issues, though.
With the following structure:

project
- Snakemake
- output/
- modules/
    - light/
        - output/ 
        - Snakemake

I run

rule run_light_submodule:
    input:
        "output/global.txt"
    output:
        "modules/light/output/final.txt"
    shell:
        'snakemake -s modules/light/Snakefile -c 1'

Which returns

LockException:
Error: Directory cannot be locked. Please make sure that no other Snakemake process is trying to create the same files in the following directory:
/home/ivanruizmanuel/Documents/git/snakemake_tests
If you are sure that no other instances of snakemake are running on this directory, the remaining lock was likely caused by a kill signal or a power loss. It can be removed with the --unlock argument.
[Wed Mar 27 15:15:13 2024]
Error in rule run_light_submodule:
    jobid: 8
    input: output/global.txt
    output: modules/light/output/final.txt
    shell:
        snakemake -s modules/light/Snakefile -c 1
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-03-27T151513.412171.snakemake.log
WorkflowError:
At least one job did not complete successfully.

It seems like the subfolder is locked by snakemake... any advice?

from snakemake.

You should pass --nolock to any "submodule" snakemake calls, as well as any config/profile args you would normally run them with

from snakemake.

In the end, I just went with the module functionality.
It's... not perfect, and does not abstract the DAG, but it's enough for now.

from snakemake.