Code Monkey home page Code Monkey logo

quaich's People

Contributors

agalitsyna avatar phlya avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar

Forkers

sansamlab lipingshu i-pletenev computationalaginglab efriman muchmorepig

quaich's Issues

view usage/verification

would it make sense to verify view/cooler compatibility before launching all rules?

also, did we decide that views should be saved w/o headers?

it seems that many rules don't support views-- should that be a future upgrade?

Runtime error at the Usage steps: "ValueError: View table is out of the bounds of chromosomes in cooler."

Hello.
I followed the Usage section in the README.
Specifically, I cloned this repository and rewrote the following four lines in config/config.yml to point at my local hg38 files.

quaich/config/config.yml

Lines 2 to 5 in a50deb3

path_genome_folder: /Users/ilya/Projects/quaich/resources/genome/
path_genome_fasta: /Users/ilya/Projects/quaich/resources/genome/hg38/hg38.fa
chromsizes: /Users/ilya/Projects/quaich/resources/genome/hg38/hg38_chromsizes.txt
view: /Users/ilya/Projects/quaich/resources/test_view.txt

Then, I proceeded to Step 4.
Dry-run snakemake --use-conda --configfile config/config.yml -n has been done successfully.
However, snakemake --use-conda --configfile config/config.yml --cores 10 raises the following error from cooltools/lib/checks.py:
ValueError: View table is out of the bounds of chromosomes in cooler.

The version of cooltools after the Usage Step 3 is 0.5.1.

I would appreciate if I could resolve this error.
Do I have to change more lines in the config files?

NameError when using coolfiles on the server

Hello! Dry-run of quaich when using local coolfiles fails with the following error:
NameError in file /home/user/quaich/workflow/Snakefile, line 109:
name 'clr' is not defined
File "/home/user/quaich/workflow/Snakefile", line 109, in
ERROR:snakemake.logging:NameError in file /home/user/quaich/workflow/Snakefile, line 109:
name 'clr' is not defined
This has been observed on the latest version of quaich, where the test case ran successfully.
The coolfiles were produced by the recent version of the distiller pipeline.
It happens without the regards of actual position of the file - even in the inputs_folder/coolers in the quaich directory - or if we use relative or complete path.
Steps to replicate - write file path into samples.tsv, delete two other .tsv files without changing any configs.

cooltools wrapper import error

Hi! I faced a problem after installing the workflow with mamba (but probably it is not the problem). And after running the pipeline got the following error on make_expected_cis step:

INFO:snakemake.logging:[Sun Mar  5 11:22:34 2023]
rule make_expected_cis:
    input: inputs/coolers/O2.mcool, resources/view.txt
    output: results/expected/O2_100000.expected.tsv
    jobid: 14
    reason: Missing output files: results/expected/O2_100000.expected.tsv
    wildcards: sample=O2, resolution=100000
    threads: 4
    resources: tmpdir=/tmp, mem_mb=32768, mem_mib=31250, runtime=60
INFO:snakemake.logging:rule make_expected_cis:
    input: inputs/coolers/O2.mcool, resources/view.txt
    output: results/expected/O2_100000.expected.tsv
    jobid: 14
    reason: Missing output files: results/expected/O2_100000.expected.tsv
    wildcards: sample=O2, resolution=100000
    threads: 4
    resources: tmpdir=/tmp, mem_mb=32768, mem_mib=31250, runtime=60
`
INFO:snakemake.logging:
Activating conda environment: .snakemake/conda/8771711dc4cd22b0e83a4f95250c3973_
WARNING:snakemake.logging:Activating conda environment: .snakemake/conda/8771711dc4cd22b0e83a4f95250c3973_
Activating conda environment: .snakemake/conda/8771711dc4cd22b0e83a4f95250c3973_
WARNING:snakemake.logging:Activating conda environment: .snakemake/conda/8771711dc4cd22b0e83a4f95250c3973_
Traceback (most recent call last):
  File "/home/dkriukov/quaich_aging/.snakemake/conda/8771711dc4cd22b0e83a4f95250c3973_/bin/cooltools", line 7, in <module>
    from cooltools.cli import cli
  File "/home/dkriukov/quaich_aging/.snakemake/conda/8771711dc4cd22b0e83a4f95250c3973_/lib/python3.9/site-packages/cooltools/__init__.py", line 18, in <module>
    from .lib import (
  File "/home/dkriukov/quaich_aging/.snakemake/conda/8771711dc4cd22b0e83a4f95250c3973_/lib/python3.9/site-packages/cooltools/lib/numutils.py", line 7, in <module>
    import numba
  File "/home/dkriukov/quaich_aging/.snakemake/conda/8771711dc4cd22b0e83a4f95250c3973_/lib/python3.9/site-packages/numba/__init__.py", line 43, in <module>
    from numba.np.ufunc import (vectorize, guvectorize, threading_layer,
  File "/home/dkriukov/quaich_aging/.snakemake/conda/8771711dc4cd22b0e83a4f95250c3973_/lib/python3.9/site-packages/numba/np/ufunc/__init__.py", line 3, in <module>
    from numba.np.ufunc.decorators import Vectorize, GUVectorize, vectorize, guvectorize
  File "/home/dkriukov/quaich_aging/.snakemake/conda/8771711dc4cd22b0e83a4f95250c3973_/lib/python3.9/site-packages/numba/np/ufunc/decorators.py", line 3, in <module>
    from numba.np.ufunc import _internal
SystemError: initialization of _internal failed without raising an exception
Traceback (most recent call last):
  File "/home/dkriukov/quaich_aging/.snakemake/scripts/tmpq0j_5pp0.wrapper.py", line 26, in <module>
    shell(
  File "/home/dkriukov/miniconda3/envs/quaich/lib/python3.10/site-packages/snakemake/shell.py", line 300, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'set -euo pipefail;  (cooltools expected-cis inputs/coolers/O2.mcool::resolutions/100000  --view resources/view.txt  --ignore-diags 0  -p 4  -o results/expected/O2_100000.expected.tsv)' returned non-zero exit status 1.
[Sun Mar  5 11:22:36 2023]
INFO:snakemake.logging:[Sun Mar  5 11:22:36 2023]
Error in rule make_expected_cis:
    jobid: 14
    input: inputs/coolers/O2.mcool, resources/view.txt
    output: results/expected/O2_100000.expected.tsv
    conda-env: /home/dkriukov/quaich_aging/.snakemake/conda/8771711dc4cd22b0e83a4f95250c3973_

ERROR:snakemake.logging:Error in rule make_expected_cis:
    jobid: 14
    input: inputs/coolers/O2.mcool, resources/view.txt
    output: results/expected/O2_100000.expected.tsv
    conda-env: /home/dkriukov/quaich_aging/.snakemake/conda/8771711dc4cd22b0e83a4f95250c3973_

Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2023-03-05T112226.404875.snakemake.log
WARNING:snakemake.logging:Complete log: .snakemake/log/2023-03-05T112226.404875.snakemake.log

I must note that this problem is not in the actual version of the repository, but in my fork. Meanwhile, I decided to write the issue here because I didn't make any changes in the actual cooltools wrapper which does perform this rule. Moreover, when I run my fork locally everything is well but after reinstallation, on the remote machine, the error above appears.

Could you help me with any advice, please?

rule dependencies

I tried creating a config that only calculates dots by turning all the do: statements to False except for the dots rules & get the following KeyError-- any guesses what might be going on?

(quaich) [fudenber@endeavour1 quaich]$ snakemake --use-conda --configfile config/config-reduced2.yml -n 
KeyError in line 176 of /project/fudenber_735/quaich_process/quaich/workflow/Snakefile: 'bed_pairs' File "/project/fudenber_735/quaich_process/quaich/workflow/Snakefile", line 176, in <module>

this error seems to resolve if I turn on everything in pileup.

Q:. does the expected calculation need to be moved around somehow in the snakefile?

Mustache environment problem 

(quaich) dkriukov@cajal:~/mice/quaich$ snakemake --use-conda --configfile config/config.yaml --cores 24
Building DAG of jobs...
Creating conda environment workflow/envs/mustache_env.yml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /home/dkriukov/mice/quaich/workflow/envs/mustache_env.yml:
Command:
mamba env create --quiet --file "/home/dkriukov/mice/quaich/.snakemake/conda/10be8f4bf2958c517f20a493be3a0e5a.yaml" --prefix "/home/dkriukov/mice/quaich/.snakemake/conda/10be8f4bf2958c517f20a493be3a0e5a"
Output:
  Package                  Version  Build       Channel                  Size
───────────────────────────────────────────────────────────────────────────────
  Install:
───────────────────────────────────────────────────────────────────────────────

  + _libgcc_mutex              0.1  main        pkgs/main/linux-64     Cached
  + ca-certificates       2020.1.1  0           pkgs/main/linux-64     Cached
  + certifi             2019.11.28  py38_0      pkgs/main/linux-64     Cached
  + ld_impl_linux-64        2.33.1  h53a641e_7  pkgs/main/linux-64     Cached
  + libedit           3.1.20181209  hc058e9b_0  pkgs/main/linux-64     Cached
  + libffi                   3.2.1  hd88cf55_4  pkgs/main/linux-64     Cached
  + libgcc-ng                9.1.0  hdf63c60_0  pkgs/main/linux-64     Cached
  + libstdcxx-ng             9.1.0  hdf63c60_0  pkgs/main/linux-64     Cached
  + ncurses                    6.2  he6710b0_0  pkgs/main/linux-64     Cached
  + openssl                 1.1.1d  h7b6447c_4  pkgs/main/linux-64     Cached
  + pip                     20.0.2  py38_1      pkgs/main/linux-64     Cached
  + python                   3.8.1  h0371630_1  pkgs/main/linux-64     Cached
  + readline                   7.0  h7b6447c_5  pkgs/main/linux-64     Cached
  + setuptools              45.2.0  py38_0      pkgs/main/linux-64     Cached
  + sqlite                  3.31.1  h7b6447c_0  pkgs/main/linux-64     Cached
  + tk                       8.6.8  hbc83047_0  pkgs/main/linux-64     Cached
  + wheel                   0.34.2  py38_0      pkgs/main/linux-64     Cached
  + xz                       5.2.4  h14c3975_4  pkgs/main/linux-64     Cached
  + zlib                    1.2.11  h7b6447c_3  pkgs/main/linux-64     Cached

  Summary:

  Install: 19 packages

  Total download: 0 B

───────────────────────────────────────────────────────────────────────────────

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Installing pip dependencies: ...working... Pip subprocess error:
  ERROR: Command errored out with exit status 1:
   command: /home/dkriukov/mice/quaich/.snakemake/conda/10be8f4bf2958c517f20a493be3a0e5a/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-f9dztx3i/pypairix/setup.py'"'"'; __file__='"'"'/tmp/pip-install-f9dztx3i/pypairix/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-f5ilsda0
       cwd: /tmp/pip-install-f9dztx3i/pypairix/
  Complete output (14 lines):
  running bdist_wheel
  running build
  running build_ext
  building 'pypairix' extension
  creating build
  creating build/temp.linux-x86_64-3.8
  creating build/temp.linux-x86_64-3.8/src
  gcc -pthread -B /home/dkriukov/mice/quaich/.snakemake/conda/10be8f4bf2958c517f20a493be3a0e5a/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -D_FILE_OFFSET_BITS=64 -D_USE_KNETFILE=1 -Isrc -I/home/dkriukov/mice/quaich/.snakemake/conda/10be8f4bf2958c517f20a493be3a0e5a/include/python3.8 -c src/bgzf.c -o build/temp.linux-x86_64-3.8/src/bgzf.o
  In file included from src/bgzf.c:31:0:
  src/bgzf.h:33:10: fatal error: zlib.h: No such file or directory
   #include <zlib.h>
            ^~~~~~~~
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for pypairix
    ERROR: Command errored out with exit status 1:
     command: /home/dkriukov/mice/quaich/.snakemake/conda/10be8f4bf2958c517f20a493be3a0e5a/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-f9dztx3i/pypairix/setup.py'"'"'; __file__='"'"'/tmp/pip-install-f9dztx3i/pypairix/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-zr4crwz0/install-record.txt --single-version-externally-managed --compile --install-headers /home/dkriukov/mice/quaich/.snakemake/conda/10be8f4bf2958c517f20a493be3a0e5a/include/python3.8/pypairix
         cwd: /tmp/pip-install-f9dztx3i/pypairix/
    Complete output (14 lines):
    running install
    running build
    running build_ext
    building 'pypairix' extension
    creating build
    creating build/temp.linux-x86_64-3.8
    creating build/temp.linux-x86_64-3.8/src
    gcc -pthread -B /home/dkriukov/mice/quaich/.snakemake/conda/10be8f4bf2958c517f20a493be3a0e5a/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -D_FILE_OFFSET_BITS=64 -D_USE_KNETFILE=1 -Isrc -I/home/dkriukov/mice/quaich/.snakemake/conda/10be8f4bf2958c517f20a493be3a0e5a/include/python3.8 -c src/bgzf.c -o build/temp.linux-x86_64-3.8/src/bgzf.o
    In file included from src/bgzf.c:31:0:
    src/bgzf.h:33:10: fatal error: zlib.h: No such file or directory
     #include <zlib.h>
              ^~~~~~~~
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/dkriukov/mice/quaich/.snakemake/conda/10be8f4bf2958c517f20a493be3a0e5a/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-f9dztx3i/pypairix/setup.py'"'"'; __file__='"'"'/tmp/pip-install-f9dztx3i/pypairix/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-zr4crwz0/install-record.txt --single-version-externally-managed --compile --install-headers /home/dkriukov/mice/quaich/.snakemake/conda/10be8f4bf2958c517f20a493be3a0e5a/include/python3.8/pypairix Check the logs for full command output.

failed

CondaEnvException: Pip failed

What I did wrong?

slurm usage for readme

worth adding slurm usage to readme?
--cluster "sbatch -A CLUSTER_ACCOUNT -t CLUSTER_TIME -p CLUSTER_PARTITION -N CLUSTER_NODES -J JOBNAME" --jobs NUM_JOBS_TO_SUBMIT

Questions and improvements

Questions that came up during the Tuesday meeting with Geoff

  • how to let users specify resources like thread?.
    • tell them to change the snakefile
    • make a separate resources file that can have cluster or local config that is then read by config yaml (some manual work), # todo ?{resources[‘expected’][‘threads’]}
    • wait for snakemake to change its format
  • loops/dots naming consistency across library
  • Test what happens with empty annotations.tsv or other files.
  • Think if pileups should be split into TAD pileups and annotation pileups?
  • should providing no samples to call_tads by default calculate them across all input datasets?
  • add some comments to yaml showing sections, move resolutions up to genome zone
  • how to enable saddles against a bigwig (e.g. GC content) or other sample?
  • insulation for multiple windows in one job
  • should insulation or call_tads be split to a separate rule for strong boundaries
  • how to do plotting?
  • write something about analyses that have a reference dataset …
  • (how) would this work for a single-cell dataset?

memory usage guidelines

It would be great to have a guideline for how much memory to request (perhaps broken down by resolution). It seems like insulation requests the highest per-thread amount by default (mem_mb=32 * 1024,, and dots the highest overall threads * 16 * 1024?

Jobs are currently failing due to oom errors with slurm sbatch defaults:
slurmstepd: error: Detected 1 oom-kill event(s) in StepId=9431588.batch. Some of your processes may have been killed by the cgroup out-of-memory handler.

Not set up as a template

The installation instructions say to use the workflow as a template, but the "Use this Template" button is not visible.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.