Thank you for this suggestion. Supporting all other types of reference genome will be a very important feature for our pipeline. I will open other reference genome in recent and let you know.

Best regard,
Yang

from nanome.

This function has been merged into master, check documents here: https://github.com/TheJacksonLaboratory/nanome/blob/master/docs/SpecificUsage.md#1-how-to-use-other-reference-genome-in-nanome

from nanome.

Thank you very much for this. I'm going to test it and I'll let you know

from nanome.

Hi, I've tested with my own data and it works well (using cpu mode for basecall). However, I noticed that it does not detect my GPU. I've manually exported cuda device (export CUDA_VISIBLE_DEVICES=0) but then, using docker I get:
"CUDA error at /builds/ofan/ont_core_cpp/ont_core/common/cuda_common.cpp:203: CUDA_ERROR_NO_DEVICE"
and using singularity:
"CUDA error at /builds/ofan/ont_core_cpp/ont_core/common/cuda_common.cpp:203: CUDA_ERROR_SYSTEM_DRIVER_MISMATCH

any hints? Outside the containers gpu guppy basecall works well.
thanx

from nanome.

Hi,
User do not need to export CUDA_VISIBLE_DEVICES again. If your computer system have GPU, the variable should be visible to pipeline, and the pipeline will automatically use GPU command options.

The CUDA_ERROR_NO_DEVICE means your system do not have GPU devices, but you have provided CUDA_VISIBLE_DEVICES. The CUDA_ERROR_SYSTEM_DRIVER_MISMATCH means your CUDA device (not V100 or P100) is not compatible with cuda library installed in default container, in this case, you have to build your container.

Would you please rerun below command to test GPU running on your platform?

nvcc --version

nvidia-smi

nextflow run main.nf -profile test,singularity

Please check which process failed, for example, if process [c1/47c145] process > Basecall failed, you can dive into folder `work/c1/47c145*' to check more detailed errors, and even rerun that command.

Also you can post the .nextflow.log and the error command log file here, in order for me to further check your problem.

from nanome.

Hi, first of all, thank you a lot for your prompt feedback. Regarding your questions/comments, I must say that I'm testing your pipeline (as I do with many) in a small workstation with the following config:

System: Kernel: 5.4.0-96-generic x86_64 bits: 64 compiler: gcc v: 9.3.0 Desktop: Cinnamon 5.2.7
wm: muffin dm: LightDM Distro: Linux Mint 20.3 Una base: Ubuntu 20.04 focal
CPU: Topology: 12-Core model: AMD Ryzen Threadripper 2920X bits: 64 type: MT MCP arch: Zen+
rev: 2 L2 cache: 6144 KiB
Graphics: Device-1: NVIDIA vendor: ASUSTeK driver: nvidia v: 510.39.01 bus ID: 42:00.0
chip ID: 10de:2484
Display: x11 server: X.Org 1.20.13 driver: nvidia
unloaded: fbdev,modesetting,nouveau,vesa resolution: 2560x1440~144Hz
OpenGL: renderer: NVIDIA GeForce RTX 3070/PCIe/SSE2 v: 4.6.0 NVIDIA 510.39.01

nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Fri_Dec_17_18:16:03_PST_2021
Cuda compilation tools, release 11.6, V11.6.55
Build cuda_11.6.r11.6/compiler.30794723_0

nvidia-smi

| NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... On | 00000000:42:00.0 On | N/A |
| 0% 25C P8 22W / 220W | 461MiB / 8192MiB | 0% Default |
| | | N/A |

nextflow run main.nf -profile test,singularity
N E X T F L O W ~ version 21.10.6
Not a valid project name: main.nf

running (without manually exporting CUDA_VISIBLE_DEVICES): nextflow run TheJacksonLaboratory/nanome -profile test,singularity

=================================
executor > local (16)
[c6/8a705d] process > EnvCheck (CIEcoli) [100%] 1 of 1 ✔
[8a/5bffcd] process > Untar (ecoli_ci_test_fast5.tar) [100%] 1 of 1 ✔
[ba/2952e6] process > Basecall (ecoli_ci_test_fast5.tar) [100%] 1 of 1 ✔
[50/4fc66d] process > QCExport (CIEcoli) [100%] 1 of 1 ✔
[cf/0eda03] process > Resquiggle (ecoli_ci_test_fast5.tar) [100%] 1 of 1 ✔
[78/1d92a2] process > Nanopolish (ecoli_ci_test_fast5.tar) [100%] 1 of 1 ✔
[15/2ab5bd] process > NplshComb (CIEcoli) [100%] 1 of 1 ✔
[f5/1d1786] process > DeepSignal (ecoli_ci_test_fast5.tar) [100%] 1 of 1 ✔
[d6/1eea5a] process > DpSigComb (CIEcoli) [100%] 1 of 1 ✔
[b8/df2702] process > Guppy (ecoli_ci_test_fast5.tar) [100%] 1 of 1 ✔
[67/cbb8b8] process > GuppyComb (CIEcoli) [100%] 1 of 1 ✔
[27/cef4dc] process > Tombo (ecoli_ci_test_fast5.tar) [100%] 1 of 1 ✔
[ca/0cc5ff] process > TomboComb (CIEcoli) [100%] 1 of 1 ✔
[a4/9ad71f] process > DeepMod (ecoli_ci_test_fast5.tar) [100%] 1 of 1 ✔
[13/e907a0] process > DpmodComb (CIEcoli) [100%] 1 of 1 ✔
[41/edfc19] process > Report (CIEcoli) [100%] 1 of 1 ✔
Completed at: 29-Jan-2022 14:48:46
Duration : 1m 26s
CPU hours : (a few seconds)
Succeeded : 16

But when checking the Basecall process command.log I can see :

CUDA_VISIBLE_DEVICES=
Detect no GPU, using CPU commandType
/usr/bin/guppy_basecaller
: Guppy Basecalling Software, (C) Oxford Nanopore Technologies, Limited. Version 5.0.16+b9fcd7b5b

So, if I add to .bashrc export CUDA_VISIBLE_DEVICES=0 (which should not be needed), I get:

[ca/10a27b] process > EnvCheck (CIEcoli) [100%] 1 of 1 ✔
[a2/ff08a2] process > Untar (ecoli_ci_test_fast5.tar) [100%] 1 of 1 ✔
[9f/bb0333] process > Basecall (ecoli_ci_test_fast5.tar) [100%] 1 of 1, failed: 1 ✔
[- ] process > QCExport -
[- ] process > Resquiggle -
[- ] process > Nanopolish -
[- ] process > NplshComb -
[- ] process > DeepSignal -
[- ] process > DpSigComb -
[41/ba8620] process > Guppy (ecoli_ci_test_fast5.tar) [100%] 1 of 1, failed: 1 ✔
[- ] process > GuppyComb -
[- ] process > Tombo -
[- ] process > TomboComb -
[- ] process > DeepMod -
[- ] process > DpmodComb -
[- ] process > Report -
Staging foreign file: https://github.com/TheJacksonLaboratory/nanome/raw/master/test_data/ecoli_ci_test_fast5.tar.gz
Pulling Singularity image docker://liuyangzzu/nanome:v1.3 [cache /home/pedro/Documents/nanopore/211203/local_singularity_cache/liuyangzzu-nanome-v1.3.img]
Staging foreign file: https://storage.googleapis.com/jax-nanopore-01-project-data/nanome-input/model.CpG.R9.4_1D.human_hx1.bn17.sn360.v0.1.7+.tar.gz
[9f/bb0333] NOTE: Process Basecall (ecoli_ci_test_fast5.tar) terminated with an error exit status (255) -- Error is ignored
[41/ba8620] NOTE: Process Guppy (ecoli_ci_test_fast5.tar) terminated with an error exit status (255) -- Error is ignored

In the Basecall process command.log I can see:

CUDA_VISIBLE_DEVICES=0
Detect GPU, using GPU commandType
/usr/bin/guppy_basecaller
: Guppy Basecalling Software, (C) Oxford Nanopore Technologies, Limited. Version 5.0.16+b9fcd7b5b

but also (CIEcoli.ecoli_ci_test_fast5.tar.Basecall.run.log):

ONT Guppy basecalling software version 5.0.16+b9fcd7b5b
config file: /opt/ont/guppy/data/dna_r9.4.1_450bps_hac.cfg
model file: /opt/ont/guppy/data/template_r9.4.1_450bps_hac.jsn
input path: ecoli_ci_test_fast5.tar.untar
save path: ecoli_ci_test_fast5.tar.basecalled
chunk size: 2000
chunks per runner: 256
minimum qscore: 9
records per file: 4000
fastq compression: ON
num basecallers: 1
gpu device: auto
kernel path:
runners per device: 4

[guppy/error] main: CUDA error at /builds/ofan/ont_core_cpp/ont_core/common/cuda_common.cpp:203: CUDA error at /builds/ofan/ont_core_cpp/ont_core/common/cuda_common.cpp:203: CUDA_ERROR_SYSTEM_DRIVER_MISMATCH
[guppy/warning] main: An error has occurred. Aborting.

Outside the container, I can easily resort to cuda for guppy gpu basecalling:

chunk size: 2000
chunks per runner: 512
minimum qscore: 8
records per file: 4000
fastq compression: ON
num basecallers: 1
gpu device: cuda:0
kernel path:
runners per device: 4
2022-01-26 15:14:54.506944 [guppy/info] crashpad_handler successfully launched.
2022-01-26 15:14:54.789242 [guppy/info] CUDA device 0 (compute 8.6) initialised, memory limit 8346468352B (7750156288B free)
2022-01-26 15:14:55.471325 [guppy/message] Found 428 fast5 files to process.
2022-01-26 15:14:55.485657 [guppy/message] Init time: 977 ms
2022-01-26 15:14:56.542819 [guppy/info] Read '597db6b0-2a75-4d22-8ebe-d354ec6bad3e' from file "FAQ28390_2b9848fd_0.fast5" has been loaded.

from nanome.

nextflow.log:

Jan-29 15:04:03.648 [main] INFO nextflow.Nextflow - dsname : CIEcoli
input : https://github.com/TheJacksonLaboratory/nanome/raw/master/test_data/ecoli_ci_test_fast5.tar.gz
genome : ecoli - [https://storage.googleapis.com/jax-nanopore-01-project-data/nanome-input/ecoli.tar.gz]

Running settings : --------
processors : 2
chrSet : NC_000913.3
dataType : ecoli
runBasecall : Yes
runNanopolish : Yes
runDeepSignal : Yes
runGuppy : Yes
runTombo : Yes
runDeepMod : Yes
runNANOME : Yes
outputRaw : true

Model summary : --------
GUPPY_BASECALL_MODEL: dna_r9.4.1_450bps_hac.cfg
DEEPSIGNAL_MODEL_DIR/DEEPSIGNAL_MODEL: model.CpG.R9.4_1D.human_hx1.bn17.sn360.v0.1.7+/bn_17.sn_360.epoch_9.ckpt
GUPPY_METHCALL_MODEL: dna_r9.4.1_450bps_modbases_5mc_hac.cfg
DEEPMOD_RNN_MODEL : rnn_conmodC_P100wd21_f7ne1u0_4/mod_train_conmodC_P100wd21_f3ne1u0

Pipeline settings : --------
Working dir : /home/pedro/Documents/nanopore/211203/work
Output dir : results
Launch dir : /home/pedro/Documents/nanopore/211203
Script dir : /home/pedro/.nextflow/assets/TheJacksonLaboratory/nanome
User : pedro
Profile : test,singularity
Config files : /home/pedro/.nextflow/assets/TheJacksonLaboratory/nanome/nextflow.config
Pipeline Release : master
Container : singularity - docker://liuyangzzu/nanome:v1.3
errorStrategy : ignore
maxRetries : 5
Jan-29 15:04:03.649 [main] INFO nextflow.Nextflow - =================================
Jan-29 15:04:04.006 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:EnvCheck matches process EnvCheck
Jan-29 15:04:04.011 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jan-29 15:04:04.011 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jan-29 15:04:04.015 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Jan-29 15:04:04.020 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=24; memory=125.7 GB; capacity=50; pollInterval=100ms; dumpInterval=5m
Jan-29 15:04:04.041 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 25; maxThreads: 1000
Jan-29 15:04:04.094 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jan-29 15:04:04.094 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jan-29 15:04:04.100 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jan-29 15:04:04.101 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jan-29 15:04:04.124 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jan-29 15:04:04.124 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jan-29 15:04:04.134 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jan-29 15:04:04.134 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jan-29 15:04:04.146 [main] DEBUG nextflow.processor.TaskProcessor - Creating combiner operator for each param(s) at index(es): [1]
Jan-29 15:04:04.153 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jan-29 15:04:04.153 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jan-29 15:04:04.156 [main] DEBUG nextflow.processor.TaskProcessor - Creating combiner operator for each param(s) at index(es): [1]
Jan-29 15:04:04.163 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jan-29 15:04:04.163 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jan-29 15:04:04.172 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jan-29 15:04:04.172 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jan-29 15:04:04.176 [main] DEBUG nextflow.processor.TaskProcessor - Creating combiner operator for each param(s) at index(es): [1, 2]
Jan-29 15:04:04.184 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jan-29 15:04:04.185 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jan-29 15:04:04.196 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jan-29 15:04:04.196 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jan-29 15:04:04.199 [main] DEBUG nextflow.processor.TaskProcessor - Creating combiner operator for each param(s) at index(es): [1, 2]
Jan-29 15:04:04.218 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jan-29 15:04:04.218 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jan-29 15:04:04.224 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jan-29 15:04:04.224 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jan-29 15:04:04.226 [main] DEBUG nextflow.processor.TaskProcessor - Creating combiner operator for each param(s) at index(es): [1]
Jan-29 15:04:04.233 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jan-29 15:04:04.234 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jan-29 15:04:04.239 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jan-29 15:04:04.239 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jan-29 15:04:04.241 [main] DEBUG nextflow.processor.TaskProcessor - Creating combiner operator for each param(s) at index(es): [1]
Jan-29 15:04:04.250 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jan-29 15:04:04.250 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jan-29 15:04:04.275 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Jan-29 15:04:04.275 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Jan-29 15:04:04.284 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: EnvCheck, Report, METEORE, QCExport, DpSigComb, Basecall, GuppyComb, Tombo, NewTool, Untar, Resquiggle, DeepMod, NewToolComb, Nanopolish, Guppy, DpmodComb, NplshComb, TomboComb, Megalodon, DeepSignal, MgldnComb
Jan-29 15:04:04.285 [main] DEBUG nextflow.extension.CH - Bridging dataflow queue=DataflowQueue(queue=[])
Jan-29 15:04:04.287 [main] DEBUG nextflow.extension.CH - Bridging dataflow queue=DataflowQueue(queue=[])
Jan-29 15:04:04.288 [main] DEBUG nextflow.extension.CH - Bridging dataflow queue=DataflowQueue(queue=[])
Jan-29 15:04:04.288 [main] DEBUG nextflow.extension.CH - Bridging dataflow queue=DataflowQueue(queue=[])
Jan-29 15:04:04.288 [main] DEBUG nextflow.Session - Ignite dataflow network (38)
Jan-29 15:04:04.295 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > EnvCheck
Jan-29 15:04:04.296 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > Untar
Jan-29 15:04:04.297 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > Basecall
Jan-29 15:04:04.297 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > QCExport
Jan-29 15:04:04.297 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > Resquiggle
Jan-29 15:04:04.297 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > Resquiggle
Jan-29 15:04:04.297 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > Nanopolish
Jan-29 15:04:04.298 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > Nanopolish
Jan-29 15:04:04.298 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NplshComb
Jan-29 15:04:04.299 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > DeepSignal
Jan-29 15:04:04.299 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > DeepSignal
Jan-29 15:04:04.299 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > DpSigComb
Jan-29 15:04:04.299 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > Guppy
Jan-29 15:04:04.299 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > Guppy
Jan-29 15:04:04.302 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > GuppyComb
Jan-29 15:04:04.304 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > Tombo
Jan-29 15:04:04.305 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > Tombo
Jan-29 15:04:04.306 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > TomboComb
Jan-29 15:04:04.306 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > DeepMod
Jan-29 15:04:04.306 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > DeepMod
Jan-29 15:04:04.306 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > DpmodComb
Jan-29 15:04:04.307 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > Report
Jan-29 15:04:04.308 [main] DEBUG nextflow.script.ScriptRunner - > Await termination
Jan-29 15:04:04.308 [main] DEBUG nextflow.Session - Session await
Jan-29 15:04:05.491 [Actor Thread 22] DEBUG n.util.BlockingThreadExecutorFactory - Thread pool name=FileTransfer; maxThreads=72; maxQueueSize=216; keepAlive=1m
Jan-29 15:04:05.496 [FileTransfer-thread-1] DEBUG nextflow.file.FilePorter - Copying foreign file https://storage.googleapis.com/jax-nanopore-01-project-data/nanome-input/ecoli.tar.gz to work dir: /home/pedro/Documents/nanopore/211203/work/stage/00/f9f02f36eeefdad0fd371c174a73c7/ecoli.tar.gz
Jan-29 15:04:05.733 [FileTransfer-thread-2] DEBUG nextflow.file.FilePorter - Copying foreign file https://storage.googleapis.com/jax-nanopore-01-project-data/nanome-input/model.CpG.R9.4_1D.human_hx1.bn17.sn360.v0.1.7+.tar.gz to work dir: /home/pedro/Documents/nanopore/211203/work/stage/17/3f93549f94c98aadaa33abe75ec1ca/model.CpG.R9.4_1D.human_hx1.bn17.sn360.v0.1.7+.tar.gz
Jan-29 15:04:05.947 [FileTransfer-thread-3] DEBUG nextflow.file.FilePorter - Copying foreign file https://github.com/TheJacksonLaboratory/nanome/raw/master/test_data/ecoli_ci_test_fast5.tar.gz to work dir: /home/pedro/Documents/nanopore/211203/work/stage/ac/7bdceb3a0dddc741c890ed4c479338/ecoli_ci_test_fast5.tar.gz
Jan-29 15:04:07.947 [Actor Thread 25] INFO nextflow.file.FilePorter - Staging foreign file: https://github.com/TheJacksonLaboratory/nanome/raw/master/test_data/ecoli_ci_test_fast5.tar.gz
Jan-29 15:04:08.206 [Actor Thread 20] INFO nextflow.container.SingularityCache - Pulling Singularity image docker://liuyangzzu/nanome:v1.3 [cache /home/pedro/Documents/nanopore/211203/local_singularity_cache/liuyangzzu-nanome-v1.3.img]
Jan-29 15:04:08.891 [Actor Thread 22] INFO nextflow.file.FilePorter - Staging foreign file: https://storage.googleapis.com/jax-nanopore-01-project-data/nanome-input/model.CpG.R9.4_1D.human_hx1.bn17.sn360.v0.1.7+.tar.gz
Jan-29 15:04:12.191 [Actor Thread 20] DEBUG nextflow.container.SingularityCache - Singularity pull complete image=docker://liuyangzzu/nanome:v1.3 path=/home/pedro/Documents/nanopore/211203/local_singularity_cache/liuyangzzu-nanome-v1.3.img
Jan-29 15:04:12.282 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Jan-29 15:04:12.286 [Task submitter] INFO nextflow.Session - [a2/ff08a2] Submitted process > Untar (ecoli_ci_test_fast5.tar)
Jan-29 15:04:12.790 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 2; name: Untar (ecoli_ci_test_fast5.tar); status: COMPLETED; exit: 0; error: -; workDir: /home/pedro/Documents/nanopore/211203/work/a2/ff08a2905c3972cc682b51622b44ed]
Jan-29 15:04:12.842 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Jan-29 15:04:12.843 [Task submitter] INFO nextflow.Session - [9f/bb0333] Submitted process > Basecall (ecoli_ci_test_fast5.tar)
Jan-29 15:04:14.226 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 3; name: Basecall (ecoli_ci_test_fast5.tar); status: COMPLETED; exit: 255; error: -; workDir: /home/pedro/Documents/nanopore/211203/work/9f/bb0333f896fb508bd348bc98bc830e]
Jan-29 15:04:14.234 [Task monitor] INFO nextflow.processor.TaskProcessor - [9f/bb0333] NOTE: Process Basecall (ecoli_ci_test_fast5.tar) terminated with an error exit status (255) -- Error is ignored
Jan-29 15:04:28.441 [Actor Thread 22] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/pedro/.nextflow/assets/TheJacksonLaboratory/nanome/utils/null1
Jan-29 15:04:28.444 [Actor Thread 22] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/pedro/.nextflow/assets/TheJacksonLaboratory/nanome/utils/validate_nanome_container.sh
Jan-29 15:04:28.444 [Actor Thread 22] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/pedro/.nextflow/assets/TheJacksonLaboratory/nanome/utils/gcf52ref.patch
Jan-29 15:04:28.444 [Actor Thread 22] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/pedro/.nextflow/assets/TheJacksonLaboratory/nanome/utils/sum_chr_mod.py
Jan-29 15:04:28.444 [Actor Thread 22] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/pedro/.nextflow/assets/TheJacksonLaboratory/nanome/utils/clean_old_basecall_in_fast5.py
Jan-29 15:04:28.444 [Actor Thread 22] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/pedro/.nextflow/assets/TheJacksonLaboratory/nanome/utils/combination_model_prediction.py
Jan-29 15:04:28.445 [Actor Thread 22] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/pedro/.nextflow/assets/TheJacksonLaboratory/nanome/utils/unify_format_for_calls.sh
Jan-29 15:04:28.445 [Actor Thread 22] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/pedro/.nextflow/assets/TheJacksonLaboratory/nanome/utils/null3
Jan-29 15:04:28.445 [Actor Thread 22] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/pedro/.nextflow/assets/TheJacksonLaboratory/nanome/utils/preindexing_check_dups.py
Jan-29 15:04:28.445 [Actor Thread 22] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/pedro/.nextflow/assets/TheJacksonLaboratory/nanome/utils/hm_cluster_predict.py
Jan-29 15:04:28.445 [Actor Thread 22] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/pedro/.nextflow/assets/TheJacksonLaboratory/nanome/utils/gcf52ref.tar.gz
Jan-29 15:04:28.445 [Actor Thread 22] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/pedro/.nextflow/assets/TheJacksonLaboratory/nanome/utils/null2
Jan-29 15:04:28.445 [Actor Thread 22] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/pedro/.nextflow/assets/TheJacksonLaboratory/nanome/utils/extract_methylation_fast5_support_dir.py
Jan-29 15:04:28.446 [Actor Thread 22] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/pedro/.nextflow/assets/TheJacksonLaboratory/nanome/utils/tombo_extract_per_read_stats.py
Jan-29 15:04:28.446 [Actor Thread 22] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/pedro/.nextflow/assets/TheJacksonLaboratory/nanome/utils/FilesSeparator.sh
Jan-29 15:04:28.446 [Actor Thread 22] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /home/pedro/.nextflow/assets/TheJacksonLaboratory/nanome/utils/FilesSeparator.py
Jan-29 15:04:28.454 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Jan-29 15:04:28.455 [Task submitter] INFO nextflow.Session - [ca/10a27b] Submitted process > EnvCheck (CIEcoli)
Jan-29 15:04:38.309 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: EnvCheck (CIEcoli); status: COMPLETED; exit: 0; error: -; workDir: /home/pedro/Documents/nanopore/211203/work/ca/10a27b848b85590e50e99c47fc4839]
Jan-29 15:04:38.310 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Process EnvCheck (CIEcoli) is unable to find [UnixPath]: /home/pedro/Documents/nanopore/211203/work/ca/10a27b848b85590e50e99c47fc4839/rerio (pattern: rerio)
Jan-29 15:04:38.378 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Process EnvCheck > Skipping output binding because one or more optional files are missing: fileoutparam<1>
Jan-29 15:04:38.453 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Jan-29 15:04:38.454 [Task submitter] INFO nextflow.Session - [41/ba8620] Submitted process > Guppy (ecoli_ci_test_fast5.tar)
Jan-29 15:04:39.813 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 4; name: Guppy (ecoli_ci_test_fast5.tar); status: COMPLETED; exit: 255; error: -; workDir: /home/pedro/Documents/nanopore/211203/work/41/ba862049350f4d29c6d65fed9a4830]
Jan-29 15:04:39.814 [Task monitor] INFO nextflow.processor.TaskProcessor - [41/ba8620] NOTE: Process Guppy (ecoli_ci_test_fast5.tar) terminated with an error exit status (255) -- Error is ignored
Jan-29 15:04:39.817 [main] DEBUG nextflow.Session - Session await > all process finished
Jan-29 15:04:39.913 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jan-29 15:04:39.923 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=2; failedCount=2; ignoredCount=2; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=10.2s; failedDuration=2.6s; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=1; peakMemory=0; ]
Jan-29 15:04:39.949 [main] DEBUG nextflow.CacheDB - Closing CacheDB done
Jan-29 15:04:39.963 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

from nanome.

Hi,
The error message CUDA_ERROR_SYSTEM_DRIVER_MISMATCH means the docker/singularity container's cuda driver is not compatible with your GPU devices. We have tested may GPU devices without problems: such as nvidia-tesla-p100, nvidia-tesla-v100, etc.

Can you go to the failed Basecall process [9f/bb0333] process > Basecall (ecoli_ci_test_fast5.tar) [100%] 1 of 1, failed: 1 ✔

cd work/9f/bb0333

Then, test if your local running guppy basecaller script can be ok?

Compare two ways of runing:

1. Below will be a local running of the script, using your local system's cuda driver

bash -x .command.sh

2. Below is the docker's calling of the script, using our default container's cuda driver

docker run -v $PWD:$PWD -w $PWD  liuyangzzu/nanome:v1.3   bash -x .command.sh

If you can run in your local runing, but failed with our container with error message CUDA_ERROR_SYSTEM_DRIVER_MISMATCH. I suggest to chose one of below two ways to run NANOME:

1. local running, check doc here: https://github.com/TheJacksonLaboratory/nanome/blob/master/docs/Usage.md#6-conda-environment-for-local-running
2. build your container by installing your GPU specific driver, based on NANOME GitHub's Dockerfile: https://github.com/TheJacksonLaboratory/nanome/blob/master/Dockerfile

from nanome.

Since the GPU specific cuda driver can work ok in the workstation, you can directly use local running NANOME:

1. Create the conda env

git clone https://github.com/TheJacksonLaboratory/nanome.git
cd nanome
conda env create --name nanome --file=environment.yml
conda activate nanome

pip install megalodon==2.4.1
npm install -g inliner
conda install -c conda-forge -c bioconda nextflow

2. Run pipeline:

# Run NANOME pipeline using local execution
conda activate nanome
nextflow run TheJacksonLaboratory/nanome\
    -profile test\
    --guppyDir [guppy-installation-directory]

from nanome.