Comments (21)
iiSeymour
commented on June 6, 2024
2
yes, we have copy thanks @edoyango and will have an update soon.
from dorado.
vellamike
commented on June 6, 2024
1
It's possible that for this particular dataset Dorado is not perfectly-tuned, chunk size/batch size etc may not be optimal - could you share some data with us and we can try and reproduce internally? FYI we are also planning to release some major performance improvements to Dorado in the coming weeks.
from dorado.
iiSeymour
commented on June 6, 2024
1
Thanks for the feedback @Kirk3gaard we have some nice speed-ups coming in the next release.
On top of that we have found that the batchsize we automatically select is particularly suboptimal on pre-ampere cards.
I am fixing this now but you might want to find a better batchsize on your RTX2080 in the meantime.
from dorado.
iiSeymour
commented on June 6, 2024
1
@edoyango I have benchmarked the dataset you provided with guppy v6.4.3, dorado v0.0.3 and the latest v0.1.0 on 1x and 4x A100. I could match the lower performance with sup relative to guppy in dorado v0.0.3 This is now resolved in v0.1.0 and we have also made significant improvements with hac.
dna_r10.4.1_e8.2_400bps_hac
| caller | 1x A100 sample/s | 4x A100 sample/s |
|---|---|---|
| guppy v6.4.3 | 2.8795e+07 | 6.361e+07 |
| dorado v0.3.0 | 3.1787e+07 | 1.1186e+08 |
| dorado v0.1.0 | 3.6346e+07 | 1.2118e+08 |
dna_r10.4.1_e8.2_400bps_sup
| caller | 1x A100 sample/s | 4x A100 sample/s |
|---|---|---|
| guppy v6.4.3 | 6.6731e+06 | 2.4761e+07 |
| dorado v0.3.0 | 5.6342e+06 | 2.2033e+07 |
| dorado v0.1.0 | 6.4252e+06 | 2.5267e+07 |
hac logs
$ guppy_basecaller -x cuda:0 -c dna_r10.4.1_e8.2_400bps_hac.cfg --num_callers 4 --gpu_runners_per_device 4 --chunks_per_runner 1280 -i $p5 -s gtest ONT Guppy basecalling software version 6.4.3+0186003, minimap2 version 2.24-r1122 config file: dna_r10.4.1_e8.2_400bps_hac.cfg model file: template_r10.4.1_e8.2_400bps_hac.jsn input path: data/edoyango/pod5 save path: gtest chunk size: 2000 chunks per runner: 1280 minimum qscore: 9 records per file: 4000 num basecallers: 4 gpu device: cuda:0 kernel path: runners per device: 4 Caller time: 24829 ms, Samples called: 714949372, samples/s: 2.87949e+07 Finishing up any open output files. Basecalling completed successfully. $ guppy_basecaller -x cuda:all -c dna_r10.4.1_e8.2_400bps_hac.cfg --num_callers 4 --gpu_runners_per_device 4 --chunks_per_runner 1280 -i $p5 -s gtest ONT Guppy basecalling software version 6.4.3+0186003, minimap2 version 2.24-r1122 config file: dna_r10.4.1_e8.2_400bps_hac.cfg model file: template_r10.4.1_e8.2_400bps_hac.jsn input path: data/edoyango/pod5 save path: gtest chunk size: 2000 chunks per runner: 1280 minimum qscore: 9 records per file: 4000 num basecallers: 4 gpu device: cuda:all kernel path: runners per device: 4 Caller time: 11240 ms, Samples called: 714949372, samples/s: 6.36076e+07 Finishing up any open output files. Basecalling completed successfully. $ dorado-0.0.3/bin/dorado basecaller [email protected] $p5 -x cuda:0 > dtest.sam [2022-12-11 15:01:22.910] [info] > Creating basecall pipeline [2022-12-11 15:01:55.242] [info] > Reads basecalled: 20000 [2022-12-11 15:01:55.242] [info] > Samples/s: 3.178710e+07 [2022-12-11 15:01:55.242] [info] > Finished $ dorado-0.0.3/bin/dorado basecaller [email protected] $p5 -x cuda:all > dtest.sam [2022-12-11 15:02:20.536] [info] > Creating basecall pipeline [2022-12-11 15:03:00.889] [info] > Reads basecalled: 20000 [2022-12-11 15:03:00.889] [info] > Samples/s: 1.118612e+08 [2022-12-11 15:03:00.889] [info] > Finished $ dorado-0.1.0/bin/dorado basecaller [email protected] $p5 -x cuda:0 > dtest.sam [2022-12-11 15:03:49.966] [info] > Creating basecall pipeline [2022-12-11 15:04:23.282] [info] > Reads basecalled: 20000 [2022-12-11 15:04:23.282] [info] > Samples/s: 3.634564e+07 [2022-12-11 15:04:23.282] [info] > Finished $ dorado-0.1.0/bin/dorado basecaller [email protected] $p5 -x cuda:all > dtest.sam [2022-12-11 15:04:32.128] [info] > Creating basecall pipeline [2022-12-11 15:05:15.882] [info] > Reads basecalled: 20000 [2022-12-11 15:05:15.882] [info] > Samples/s: 1.211829e+08 [2022-12-11 15:05:15.882] [info] > Finished
sup logs
$ guppy_basecaller -x cuda:0 -c dna_r10.4.1_e8.2_400bps_sup.cfg --num_callers 4 --gpu_runners_per_device 4 --chunks_per_runner 512 -i $p5 -s gtest ONT Guppy basecalling software version 6.4.3+0186003, minimap2 version 2.24-r1122 config file: dna_r10.4.1_e8.2_400bps_sup.cfg model file: template_r10.4.1_e8.2_400bps_sup.jsn input path: data/edoyango/pod5 save path: gtest chunk size: 2000 chunks per runner: 512 minimum qscore: 10 records per file: 4000 num basecallers: 4 gpu device: cuda:0 kernel path: runners per device: 4 Caller time: 107139 ms, Samples called: 714949372, samples/s: 6.6731e+06 Finishing up any open output files. Basecalling completed successfully. $ guppy_basecaller -x cuda:all -c dna_r10.4.1_e8.2_400bps_sup.cfg --num_callers 4 --gpu_runners_per_device 4 --chunks_per_runner 512 -i $p5 -s gtest ONT Guppy basecalling software version 6.4.3+0186003, minimap2 version 2.24-r1122 config file: dna_r10.4.1_e8.2_400bps_sup.cfg model file: template_r10.4.1_e8.2_400bps_sup.jsn input path: data/edoyango/pod5 save path: gtest chunk size: 2000 chunks per runner: 512 minimum qscore: 10 records per file: 4000 num basecallers: 4 gpu device: cuda:all kernel path: runners per device: 4 Caller time: 28874 ms, Samples called: 714949372, samples/s: 2.4761e+07 Finishing up any open output files. Basecalling completed successfully. $ dorado-0.0.3/bin/dorado basecaller [email protected] $p5 -x cuda:0 > dtest.sam [2022-12-11 14:42:03.019] [info] > Creating basecall pipeline [2022-12-11 14:44:31.746] [info] > Reads basecalled: 20000 [2022-12-11 14:44:31.746] [info] > Samples/s: 5.634152e+06 [2022-12-11 14:44:31.746] [info] > Finished $ dorado-0.0.3/bin/dorado basecaller [email protected] $p5 -x cuda:all > dtest.sam [2022-12-11 14:44:40.465] [info] > Creating basecall pipeline [2022-12-11 14:45:50.835] [info] > Reads basecalled: 20000 [2022-12-11 14:45:50.835] [info] > Samples/s: 2.203258e+07 [2022-12-11 14:45:50.835] [info] > Finished $ dorado-0.1.0/bin/dorado basecaller [email protected] $p5 -x cuda:0 > dtest.sam [2022-12-11 14:48:16.175] [info] > Creating basecall pipeline [2022-12-11 14:50:22.279] [info] > Reads basecalled: 20000 [2022-12-11 14:50:22.279] [info] > Samples/s: 6.425246e+06 [2022-12-11 14:50:22.279] [info] > Finished $ dorado-0.1.0/bin/dorado basecaller [email protected] $p5 -x cuda:all > dtest.sam [2022-12-11 14:46:41.796] [info] > Creating basecall pipeline [2022-12-11 14:47:51.576] [info] > Reads basecalled: 20000 [2022-12-11 14:47:51.576] [info] > Samples/s: 2.526713e+07 [2022-12-11 14:47:51.576] [info] > Finished
@Kirk3gaard I saw you are still seeing suboptimal results with RTX2080. Can you provide the full guppy & dorado commands and output, please?
from dorado.
edoyango
commented on June 6, 2024
1
Hi @iiSeymour,
Thanks for the update. Can confirm v0.1.0 comes with a performance boost. However, the group of researchers I'm helping are mostly interested in the SUP models, and guppy seems to edge out (at least for 1x A100). Can we expect any improvements in dorado for SUP for our case in future releases?
Can I also ask whether dorado default parameters are tuned for A30s as well? Or should we play with the batchsize option. We have many more of A30 GPUs on our cluster than A100s.
from dorado.
edoyango
commented on June 6, 2024
1
I did some testing on our A30s yesterday and found that building from source speeds up dorado with SUP by a fair bit
dorado v0.1.1 using prebuilt binaries
Samples/s: 2.956463e+06
dorado v0.1.1 using binaries built from source
Samples/s: 3.673492e+06
guppy v6.2.1
samples/s: 3.16747e+06
I don't observe this speedup with the A100s though. Do you know why there's a discrepancy only for the A30s?
The dorado command is (run in a Slurm environment)
srun time dorado basecaller models/[email protected] test-pods > /dev/null
My build command:
cmake -S . -B cmake-build -DDORADO_USING_OLD_CPP_ABI=True -DCMAKE_BUILD_TYPE=Release
from dorado.
vellamike
commented on June 6, 2024
Hi @edoyango
We haven't benchmarked on A30, but somewhat surprised that you are observing Dorado to be slower on A100. I have some suggestions:
- Only use POD5
- Could you try our latest pre-built dorado release - v0.0.2
- What is the absolute speed you are observing on A100 for the HAC as well as SUP model?
- What is the read length distribution like for your dataset? How long are your reads, with what sort of variance?
Best wishes,
Mike
from dorado.
edoyango
commented on June 6, 2024
Hi @vellamike,
thanks for the suggestions. Below are results with 1x A100. Dorado is using pod5 files and is built from the v0.0.2 release.
Looks like with the HAC model both guppy and dorado perform similarly.
I'm not sure how to comment on the input dataset - I don't have a biological science background and the researcher most familiar with the data is not around to answer.
SUP
dorado basecaller ${DORADO_ROOT}/models/[email protected] test-pods > /dev/null
...
[2022-11-11 10:28:39.841] [info] > Samples/s: 5.459015e+06
...guppy_basecaller -x 'cuda:0' -c dna_r10.4.1_e8.2_400bps_sup.cfg --num_callers 4 --gpu_runners_per_device 4 --chunks_per_runner 512 -i test-fast5 -s guppy-A100-fast5-results
...
Caller time: 106640 ms, Samples called: 714949372, samples/s: 6.70433e+06
...HAC
dorado basecaller ${DORADO_ROOT}/models/[email protected] test-pods > /dev/null
...
[2022-11-11 10:48:55.355] [info] > Samples/s: 3.009059e+07
...guppy_basecaller -x 'cuda:0' -c dna_r10.4.1_e8.2_400bps_hac.cfg --num_callers 4 --gpu_runners_per_device 4 --chunks_per_runner 1280 -i test-fast5 -s guppy-A100-fast5-results
...
Caller time: 24001 ms, Samples called: 714949372, samples/s: 2.97883e+07
...from dorado.
vellamike
commented on June 6, 2024
@edoyango are you using the released binaries, or compiling yourself from source?
from dorado.
edoyango
commented on June 6, 2024
@vellamike I'm compiling from source - I might've missed it, but I couldn't find any prebuilt binaries on the repo. The link you posted before only had the source code in .zip and .tar.gz.
from dorado.
vellamike
commented on June 6, 2024
Here are links for our released binaries (in retrospect, seems obvious we should be putting them in the Release notes...)
https://github.com/nanoporetech/dorado#installation
dorado-0.0.2-linux-x64
dorado-0.0.2-osx-arm64
dorado-0.0.2-win64
I don't expect these to be faster - but I'd like to exclude the possibility that your compilation environment is responsible for the delta.
from dorado.
edoyango
commented on June 6, 2024
Ah I brainfarted and assumed those were links to source code (which wouldn't make sense because they have OS-arch in the name)
Anyway, I'll try them out on Monday when I'm back at work.
from dorado.
edoyango
commented on June 6, 2024
Hi @vellamike, results using the prebuilt binaries:
SUP:
dorado basecaller ${DORADO_ROOT}/models/[email protected] test-pods > /dev/null
...
[2022-11-14 08:53:43.960] [info] > Samples/s: 5.635749e+06
HAC:
dorado basecaller ${DORADO_ROOT}/models/[email protected] test-pods > /dev/null
...
[2022-11-14 08:59:27.995] [info] > Samples/s: 3.026241e+07
A small improvement.
Some inormation about the input data from the researcher:
The DNA was obtained from a CTAB extraction without additional shearing or size selection (CTAB cuts the DNA already). So the reads have a N50 of 7 kb with a tail tapering off at about 30kb. It would be considered as short for genomic DNA, but nothing extreme (cDNA would be much shorter, and it’s definitely not ultra-long).
from dorado.
edoyango
commented on June 6, 2024
@vellamike Here is a link with a small sample of the original fast5 pea dna data:
https://wehieduau-my.sharepoint.com/:f:/g/personal/yang_e_wehi_edu_au/EofQaH5qQhVNsuupH2npVn0Bj82LCOKwrY3MM9gU32_uDQ?e=Yu4b3q
The link will expire next friday. Let me know if you have issues downloading it.
When testing, I converted the fast5 files to pod5 with the pod5-format-tools package.
from dorado.
edoyango
commented on June 6, 2024
Hi @vellamike
Touching base regarding the test dataset I provided over a week ago. Has your team managed to get a copy?
from dorado.
Kirk3gaard
commented on June 6, 2024
I have also found dorado (.0.0.3) to be slower than guppy (6.4.2) on our RTX2080 GPUs (https://twitter.com/kirk3gaard/status/1597192469212381185). However, it still ran very well and right out of the box which is a great plus for an early release :D.
pod5 and fast5 file used in my test including the basecalled fastqs are available here: https://www.dropbox.com/sh/v7wq5r8xnrri7ti/AAA3CbwW6uutJQl-1UJWtv65a?dl=0
The data is from the zymo mock sequenced on the promethion w. LSK114.
Looking forward to see the performance upgrades for dorado.
from dorado.
Kirk3gaard
commented on June 6, 2024
Hi @iiSeymour
Here are the full commands and the output from the commands.
Best regards
Rasmus
dorado -v
0.1.0+4b0e9a6
time dorado basecaller models/[email protected] pod5s/ > calls.sam
[2022-12-06 14:47:03.608] [info] > Creating basecall pipeline
[2022-12-06 14:51:46.040] [info] > Reads basecalled: 10000
[2022-12-06 14:51:46.040] [info] > Samples/s: 3.256044e+06
[2022-12-06 14:51:46.040] [info] > Finished
real 4m43.776s
user 9m13.645s
sys 1m39.643s
time guppy_basecaller -i fast5/ -s out_guppy -c dna_r10.4.1_e8.2_400bps_sup.cfg --device 'cuda:all'
ONT Guppy basecalling software version 6.4.2+97a7f06, minimap2 version 2.24-r1122
config file: /opt/ont/guppy/data/dna_r10.4.1_e8.2_400bps_sup.cfg
model file: /opt/ont/guppy/data/template_r10.4.1_e8.2_400bps_sup.jsn
input path: fast5/
save path: out_guppy
chunk size: 2000
chunks per runner: 208
minimum qscore: 10
records per file: 4000
num basecallers: 4
gpu device: cuda:all
kernel path:
runners per device: 12
Use of this software is permitted solely under the terms of the end user license agreement (EULA).
By running, copying or accessing this software, you are demonstrating your acceptance of the EULA.
The EULA may be found in /opt/ont/guppy/bin
Found 1 input read file to process.
Init time: 6010 ms
0% 10 20 30 40 50 60 70 80 90 100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 246441 ms, Samples called: 879732046, samples/s: 3.56975e+06
Finishing up any open output files.
Basecalling completed successfully.
real 4m13.453s
user 6m51.728s
sys 2m47.935s
from dorado.
iiSeymour
commented on June 6, 2024
@edoyango v0.1.0 will do a sweep over a selection of batchsizes prior to basecalling and selects the optimal (you can see the selected batchsize by adding -v).
Interesting finding when building from source - I suspect this might be down to using a newer cuda toolkit / cublas than we are using. Can you post the output from the build step please?
from dorado.
edoyango
commented on June 6, 2024
From Cmake:
-- The C compiler identification is GNU 9.1.0
-- The CXX compiler identification is GNU 9.1.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /stornext/System/data/apps/gcc/gcc-9.1.0/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /stornext/System/data/apps/gcc/gcc-9.1.0/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Submodule update
-- Found HDF5: hdf5-static;hdf5_cpp-static (found version "1.12.2") found components: C CXX HL
-- Using prebuilt Koi from Box
-- Found koi_lib
-- KOI_DIR is /home/users/allstaff/yang.e/vast-scratch/dorado-0.1.1-src-generic/dorado/3rdparty/koi_lib/libkoi-0.1.6-Linux
-- KOI_INCLUDE is /home/users/allstaff/yang.e/vast-scratch/dorado-0.1.1-src-generic/dorado/3rdparty/koi_lib/libkoi-0.1.6-Linux/include
koi.h
-- Found pod5-0.0.41-Linux
-- Found torch-no-cxx11-abi-1.12.1-Linux
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /stornext/System/data/nvidia/cuda/cuda-11.4 (found version "11.4")
-- The CUDA compiler identification is NVIDIA 11.4.120
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /stornext/System/data/nvidia/cuda/cuda-11.4/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Caffe2: CUDA detected: 11.4
-- Caffe2: CUDA nvcc is: /stornext/System/data/nvidia/cuda/cuda-11.4/bin/nvcc
-- Caffe2: CUDA toolkit directory: /stornext/System/data/nvidia/cuda/cuda-11.4
-- Caffe2: Header version is: 11.4
-- Found CUDNN: /home/users/allstaff/yang.e/vast-scratch/dorado-0.1.1-src-generic/dorado/3rdparty/fake_cudnn
-- Found cuDNN: v8.. (include: /home/users/allstaff/yang.e/vast-scratch/dorado-0.1.1-src-generic/dorado/3rdparty/fake_cudnn, library: /home/users/allstaff/yang.e/vast-scratch/dorado-0.1.1-src-generic/dorado/3rdparty/fake_cudnn)
-- /stornext/System/data/nvidia/cuda/cuda-11.4/lib64/libnvrtc.so shorthash is edac1e2e
-- Automatic GPU detection failed. Building for common architectures.
-- Autodetected CUDA architecture(s): 3.5;5.0;5.2;6.0;6.1;7.0;7.5;8.0;8.6;8.6+PTX
-- Added CUDA NVCC flags for: -gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_86,code=compute_86
-- Found Torch: /home/users/allstaff/yang.e/vast-scratch/dorado-0.1.1-src-generic/dorado/3rdparty/torch-no-cxx11-abi-1.12.1-Linux/libtorch/lib/libtorch.so
-- Found OpenSSL: /stornext/Home/data/allstaff/y/yang.e/spack/opt/spack/linux-centos7-broadwell/gcc-9.1.0/openssl-1.1.1s-4q6f4x5jg3ltw34arjere2ljj3krrup4/lib/libcrypto.a (found version "1.1.1s")
-- Building htslib
-- Done Building htslib
-- Using CMake version 3.25.0
CMake Deprecation Warning at dorado/3rdparty/elzip/extlibs/minizip/CMakeLists.txt:56 (cmake_policy):
The OLD behavior for policy CMP0074 will be removed from a future version
of CMake.
The cmake-policies(7) manual explains that the OLD behaviors of all
policies are deprecated and that a policy should be set to OLD only under
specific short-term circumstances. Projects should be ported to the NEW
behavior and not rely on setting a policy to OLD.
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for inttypes.h
-- Looking for inttypes.h - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of off64_t
-- Check size of off64_t - failed
-- Looking for fseeko
-- Looking for fseeko - found
-- Using ZLIB 1.2.13
-- Found PkgConfig: /stornext/Home/data/allstaff/y/yang.e/spack/opt/spack/linux-centos7-broadwell/gcc-9.1.0/pkgconf-1.8.0-orqhkxpsja4idkjvdvsxjygmvo6tk2pf/bin/pkg-config (found version "1.8.0")
-- Checking for module 'openssl'
-- Found openssl, version 1.1.1s
-- Using OpenSSL 1.1.1s
-- Character encoding support requires iconv
-- The following features have been enabled:
* MZ_COMPAT, Enables compatibility layer
* MZ_ZLIB, Enables ZLIB compression
* MZ_FETCH_LIBS, Enables fetching third-party libraries if not found
* MZ_OPENSSL, Enables OpenSSL for encryption
* MZ_LIBBSD, Build with libbsd for crypto random
-- The following features have been disabled:
* MZ_BZIP2, Enables BZIP2 compression
* MZ_LZMA, Enables LZMA & XZ compression
* MZ_ZSTD, Enables ZSTD compression
* MZ_LIBCOMP, Enables Apple compression
* MZ_FORCE_FETCH_LIBS, Enables fetching third-party libraries always
* MZ_PKCRYPT, Enables PKWARE traditional encryption
* MZ_WZAES, Enables WinZIP AES encryption
* MZ_SIGNING, Enables zip signing support
* MZ_ICONV, Enables iconv string encoding conversion library
* MZ_COMPRESS_ONLY, Only support compression
* MZ_DECOMPRESS_ONLY, Only support decompression
* MZ_FILE32_API, Builds using posix 32-bit file api
* MZ_BUILD_TESTS, Builds minizip test executable
* MZ_BUILD_UNIT_TESTS, Builds minizip unit test project
* MZ_BUILD_FUZZ_TESTS, Builds minizip fuzzer executables
* MZ_CODE_COVERAGE, Builds with code coverage flags
Setting warning flags
-- Performing Test WOLD_STYLE_CAST
-- Performing Test WOLD_STYLE_CAST - Success
-- Performing Test WSHADOW
-- Performing Test WSHADOW - Success
-- CUDA toolkit dir is /stornext/System/data/nvidia/cuda/cuda-11.4
-- Building version 1.0.3
-- Unable to determine HDF5 C flags from HDF5 wrapper.
-- Could NOT find HDF5 (missing: HDF5_LIBRARIES HDF5_HL_LIBRARIES) (found suitable version "1.12.2", minimum required is "1.8.16")
-- Found zstd: /usr/lib64/libzstd.so (found suitable version "1.5.0", minimum required is "1.3.1")
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success
-- Not building hdf5 testing as hdf5 not found
-- Configuring done
CMake Warning at dorado/3rdparty/torch-no-cxx11-abi-1.12.1-Linux/libtorch/share/cmake/Caffe2/public/cuda.cmake:347 (set_property):
Target "dorado_lib" requests linking to directory
"/home/users/allstaff/yang.e/vast-scratch/dorado-0.1.1-src-generic/dorado/3rdparty/fake_cudnn".
Targets may link only to libraries. CMake is dropping the item.
Call Stack (most recent call first):
dorado/3rdparty/torch-no-cxx11-abi-1.12.1-Linux/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
dorado/3rdparty/torch-no-cxx11-abi-1.12.1-Linux/libtorch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
cmake/Torch.cmake:41 (find_package)
CMakeLists.txt:18 (include)
CMake Warning at dorado/3rdparty/torch-no-cxx11-abi-1.12.1-Linux/libtorch/share/cmake/Caffe2/public/cuda.cmake:347 (set_property):
Target "dorado" requests linking to directory
"/home/users/allstaff/yang.e/vast-scratch/dorado-0.1.1-src-generic/dorado/3rdparty/fake_cudnn".
Targets may link only to libraries. CMake is dropping the item.
Call Stack (most recent call first):
dorado/3rdparty/torch-no-cxx11-abi-1.12.1-Linux/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
dorado/3rdparty/torch-no-cxx11-abi-1.12.1-Linux/libtorch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
cmake/Torch.cmake:41 (find_package)
CMakeLists.txt:18 (include)
CMake Warning (dev) at dorado/3rdparty/edlib/CMakeLists.txt:54 (add_library):
Policy CMP0063 is not set: Honor visibility properties for all target
types. Run "cmake --help-policy CMP0063" for policy details. Use the
cmake_policy command to set the policy and suppress this warning.
Target "edlib" of type "STATIC_LIBRARY" has the following visibility
properties set for CXX:
CXX_VISIBILITY_PRESET
VISIBILITY_INLINES_HIDDEN
For compatibility CMake is not honoring them for this target.
This warning is for project developers. Use -Wno-dev to suppress it.
CMake Warning at dorado/3rdparty/torch-no-cxx11-abi-1.12.1-Linux/libtorch/share/cmake/Caffe2/public/cuda.cmake:347 (set_property):
Target "dorado_tests" requests linking to directory
"/home/users/allstaff/yang.e/vast-scratch/dorado-0.1.1-src-generic/dorado/3rdparty/fake_cudnn".
Targets may link only to libraries. CMake is dropping the item.
Call Stack (most recent call first):
dorado/3rdparty/torch-no-cxx11-abi-1.12.1-Linux/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
dorado/3rdparty/torch-no-cxx11-abi-1.12.1-Linux/libtorch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
cmake/Torch.cmake:41 (find_package)
CMakeLists.txt:18 (include)
-- Generating done
-- Build files have been written to: /home/users/allstaff/yang.e/vast-scratch/dorado-0.1.1-src-generic/cmake-build
From make:
[ 1%] Creating directories for 'htslib_project'
[ 2%] No download step for 'htslib_project'
[ 3%] No update step for 'htslib_project'
[ 4%] No patch step for 'htslib_project'
[ 5%] Performing configure step for 'htslib_project'
checking for gcc... /stornext/System/data/apps/gcc/gcc-9.1.0/bin/gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether /stornext/System/data/apps/gcc/gcc-9.1.0/bin/gcc accepts -g... yes
checking for /stornext/System/data/apps/gcc/gcc-9.1.0/bin/gcc option to accept ISO C89... none needed
checking for ranlib... /stornext/System/data/apps/gcc/gcc-9.1.0/bin/gcc-ranlib
checking for grep that handles long lines and -e... /usr/bin/grep
checking for C compiler warning flags... -Wall
checking whether C compiler accepts -mssse3 -mpopcnt -msse4.1... yes
checking whether C compiler accepts -mavx2... yes
checking whether C compiler accepts -mavx512f... yes
checking whether C compiler supports ARM Neon... no
checking for pkg-config... /stornext/Home/data/allstaff/y/yang.e/spack/opt/spack/linux-centos7-broadwell/gcc-9.1.0/pkgconf-1.8.0-orqhkxpsja4idkjvdvsxjygmvo6tk2pf/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking shared library type for unknown-Linux... plain .so
checking whether the compiler accepts -fvisibility=hidden... yes
checking how to run the C preprocessor... /stornext/System/data/apps/gcc/gcc-9.1.0/bin/gcc -E
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for stdlib.h... (cached) yes
checking for unistd.h... (cached) yes
checking for sys/param.h... yes
checking for getpagesize... yes
checking for working mmap... yes
checking for gmtime_r... yes
checking for fsync... yes
checking for drand48... yes
checking for srand48_deterministic... no
checking whether fdatasync is declared... yes
checking for fdatasync... yes
checking for library containing log... -lm
checking for zlib.h... yes
checking for inflate in -lz... yes
checking for library containing recv... none required
checking whether htscodecs files are present... yes
checking for libdeflate.h... no
checking for libdeflate_deflate_compress in -ldeflate... no
checking for library containing regcomp... none required
checking whether PTHREAD_MUTEX_RECURSIVE is declared... yes
configure: creating ./config.status
config.status: creating config.mk
config.status: creating htslib.pc.tmp
config.status: creating config.h
config.status: config.h is unchanged
config.status: linking htscodecs_bundled.mk to htscodecs.mk
[ 6%] Performing build step for 'htslib_project'
[ 7%] No install step for 'htslib_project'
[ 8%] Completed 'htslib_project'
[ 8%] Built target htslib_project
[ 10%] Creating directories for 'streamvbyte'
[ 11%] No download step for 'streamvbyte'
[ 12%] No patch step for 'streamvbyte'
[ 13%] Performing configure step for 'streamvbyte'
-- No build type selected, default to Release
-- The C compiler identification is GNU 9.1.0
-- The CXX compiler identification is GNU 9.1.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /stornext/System/data/apps/gcc/gcc-9.1.0/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /stornext/System/data/apps/gcc/gcc-9.1.0/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Deprecation Warning at CMakeLists.txt:9 (cmake_policy):
The OLD behavior for policy CMP0065 will be removed from a future version
of CMake.
The cmake-policies(7) manual explains that the OLD behaviors of all
policies are deprecated and that a policy should be set to OLD only under
specific short-term circumstances. Projects should be ported to the NEW
behavior and not rely on setting a policy to OLD.
-- CMAKE_SYSTEM_PROCESSOR: x86_64
-- CMAKE_BUILD_TYPE: Release
-- CMAKE_C_COMPILER: /stornext/System/data/apps/gcc/gcc-9.1.0/bin/gcc
-- CMAKE_C_FLAGS: "-std=c99"
-- CMAKE_C_FLAGS_DEBUG: -g
-- CMAKE_C_FLAGS_RELEASE: -O3 -DNDEBUG
-- Configuring done
-- Generating done
-- Build files have been written to: /home/users/allstaff/yang.e/vast-scratch/dorado-0.1.1-src-generic/cmake-build/streamvbyte/src/streamvbyte-build
[ 14%] Performing build step for 'streamvbyte'
[ 4%] Building C object CMakeFiles/streamvbyte_static.dir/src/streamvbyte_encode.c.o
[ 8%] Building C object CMakeFiles/streamvbyte_static.dir/src/streamvbyte_decode.c.o
[ 12%] Building C object CMakeFiles/streamvbyte_static.dir/src/streamvbyte_zigzag.c.o
[ 16%] Building C object CMakeFiles/streamvbyte_static.dir/src/streamvbytedelta_encode.c.o
[ 20%] Building C object CMakeFiles/streamvbyte_static.dir/src/streamvbytedelta_decode.c.o
[ 25%] Building C object CMakeFiles/streamvbyte_static.dir/src/streamvbyte_0124_encode.c.o
[ 29%] Building C object CMakeFiles/streamvbyte_static.dir/src/streamvbyte_0124_decode.c.o
[ 33%] Linking C static library libstreamvbyte_static.a
[ 33%] Built target streamvbyte_static
[ 37%] Building C object CMakeFiles/streamvbyte.dir/src/streamvbyte_encode.c.o
[ 41%] Building C object CMakeFiles/streamvbyte.dir/src/streamvbyte_decode.c.o
[ 45%] Building C object CMakeFiles/streamvbyte.dir/src/streamvbyte_zigzag.c.o
[ 50%] Building C object CMakeFiles/streamvbyte.dir/src/streamvbytedelta_encode.c.o
[ 54%] Building C object CMakeFiles/streamvbyte.dir/src/streamvbytedelta_decode.c.o
[ 58%] Building C object CMakeFiles/streamvbyte.dir/src/streamvbyte_0124_encode.c.o
[ 62%] Building C object CMakeFiles/streamvbyte.dir/src/streamvbyte_0124_decode.c.o
[ 66%] Linking C shared library libstreamvbyte.so
[ 66%] Built target streamvbyte
[ 70%] Building C object CMakeFiles/example.dir/example.c.o
[ 75%] Linking C executable example
Using built-in specs.
COLLECT_GCC=/stornext/System/data/apps/gcc/gcc-9.1.0/bin/gcc
COLLECT_LTO_WRAPPER=/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ./gcc-9.1.0/configure --prefix=/stornext/System/data/apps/gcc/gcc-9.1.0 --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,fortran,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --disable-multilib --build=x86_64-redhat-linux
Thread model: posix
gcc version 9.1.0 (GCC)
COMPILER_PATH=/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/:/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/:/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/
LIBRARY_PATH=/stornext/System/data/apps/gcc/gcc-9.1.0/lib64/../lib64/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/../lib64/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/stornext/System/data/tools/openSSL/openSSL-1.1.1n/lib/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib64/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/:/stornext/System/data/nvidia/cuda/cuda-11.4/lib64/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-std=c99' '-O3' '-D' 'NDEBUG' '-v' '-rdynamic' '-o' 'example' '-std=c99' '-fPIC' '-Wextra' '-Wpedantic' '-Wshadow' '-O3' '-g' '-mtune=generic' '-march=x86-64'
/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/collect2 -plugin /stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/liblto_plugin.so -plugin-opt=/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/lto-wrapper -plugin-opt=-fresolution=/vast/scratch/users/yang.e/tmp/cc2DhIaH.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --eh-frame-hdr --hash-style=gnu -m elf_x86_64 -export-dynamic -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o example /lib/../lib64/crt1.o /lib/../lib64/crti.o /stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/crtbegin.o -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib64/../lib64 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib/../lib64 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/stornext/System/data/tools/openSSL/openSSL-1.1.1n/lib -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib64 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib -L/stornext/System/data/nvidia/cuda/cuda-11.4/lib64 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/../../.. CMakeFiles/example.dir/example.c.o libstreamvbyte_static.a -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/crtend.o /lib/../lib64/crtn.o
COLLECT_GCC_OPTIONS='-std=c99' '-O3' '-D' 'NDEBUG' '-v' '-rdynamic' '-o' 'example' '-std=c99' '-fPIC' '-Wextra' '-Wpedantic' '-Wshadow' '-O3' '-g' '-mtune=generic' '-march=x86-64'
[ 75%] Built target example
[ 79%] Building C object CMakeFiles/perf.dir/tests/perf.c.o
[ 83%] Linking C executable perf
Using built-in specs.
COLLECT_GCC=/stornext/System/data/apps/gcc/gcc-9.1.0/bin/gcc
COLLECT_LTO_WRAPPER=/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ./gcc-9.1.0/configure --prefix=/stornext/System/data/apps/gcc/gcc-9.1.0 --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,fortran,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --disable-multilib --build=x86_64-redhat-linux
Thread model: posix
gcc version 9.1.0 (GCC)
COMPILER_PATH=/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/:/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/:/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/
LIBRARY_PATH=/stornext/System/data/apps/gcc/gcc-9.1.0/lib64/../lib64/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/../lib64/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/stornext/System/data/tools/openSSL/openSSL-1.1.1n/lib/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib64/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/:/stornext/System/data/nvidia/cuda/cuda-11.4/lib64/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-std=c99' '-O3' '-D' 'NDEBUG' '-v' '-rdynamic' '-o' 'perf' '-std=c99' '-fPIC' '-Wextra' '-Wpedantic' '-Wshadow' '-O3' '-g' '-mtune=generic' '-march=x86-64'
/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/collect2 -plugin /stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/liblto_plugin.so -plugin-opt=/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/lto-wrapper -plugin-opt=-fresolution=/vast/scratch/users/yang.e/tmp/ccopBz38.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --eh-frame-hdr --hash-style=gnu -m elf_x86_64 -export-dynamic -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o perf /lib/../lib64/crt1.o /lib/../lib64/crti.o /stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/crtbegin.o -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib64/../lib64 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib/../lib64 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/stornext/System/data/tools/openSSL/openSSL-1.1.1n/lib -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib64 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib -L/stornext/System/data/nvidia/cuda/cuda-11.4/lib64 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/../../.. CMakeFiles/perf.dir/tests/perf.c.o libstreamvbyte_static.a -lm -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/crtend.o /lib/../lib64/crtn.o
COLLECT_GCC_OPTIONS='-std=c99' '-O3' '-D' 'NDEBUG' '-v' '-rdynamic' '-o' 'perf' '-std=c99' '-fPIC' '-Wextra' '-Wpedantic' '-Wshadow' '-O3' '-g' '-mtune=generic' '-march=x86-64'
[ 83%] Built target perf
[ 87%] Building C object CMakeFiles/writeseq.dir/tests/writeseq.c.o
[ 91%] Linking C executable writeseq
Using built-in specs.
COLLECT_GCC=/stornext/System/data/apps/gcc/gcc-9.1.0/bin/gcc
COLLECT_LTO_WRAPPER=/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ./gcc-9.1.0/configure --prefix=/stornext/System/data/apps/gcc/gcc-9.1.0 --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,fortran,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --disable-multilib --build=x86_64-redhat-linux
Thread model: posix
gcc version 9.1.0 (GCC)
COMPILER_PATH=/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/:/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/:/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/
LIBRARY_PATH=/stornext/System/data/apps/gcc/gcc-9.1.0/lib64/../lib64/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/../lib64/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/stornext/System/data/tools/openSSL/openSSL-1.1.1n/lib/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib64/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/:/stornext/System/data/nvidia/cuda/cuda-11.4/lib64/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-std=c99' '-O3' '-D' 'NDEBUG' '-v' '-rdynamic' '-o' 'writeseq' '-std=c99' '-fPIC' '-Wextra' '-Wpedantic' '-Wshadow' '-O3' '-g' '-mtune=generic' '-march=x86-64'
/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/collect2 -plugin /stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/liblto_plugin.so -plugin-opt=/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/lto-wrapper -plugin-opt=-fresolution=/vast/scratch/users/yang.e/tmp/ccUakFrA.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --eh-frame-hdr --hash-style=gnu -m elf_x86_64 -export-dynamic -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o writeseq /lib/../lib64/crt1.o /lib/../lib64/crti.o /stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/crtbegin.o -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib64/../lib64 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib/../lib64 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/stornext/System/data/tools/openSSL/openSSL-1.1.1n/lib -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib64 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib -L/stornext/System/data/nvidia/cuda/cuda-11.4/lib64 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/../../.. CMakeFiles/writeseq.dir/tests/writeseq.c.o libstreamvbyte_static.a -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/crtend.o /lib/../lib64/crtn.o
COLLECT_GCC_OPTIONS='-std=c99' '-O3' '-D' 'NDEBUG' '-v' '-rdynamic' '-o' 'writeseq' '-std=c99' '-fPIC' '-Wextra' '-Wpedantic' '-Wshadow' '-O3' '-g' '-mtune=generic' '-march=x86-64'
[ 91%] Built target writeseq
[ 95%] Building C object CMakeFiles/unit.dir/tests/unit.c.o
[100%] Linking C executable unit
Using built-in specs.
COLLECT_GCC=/stornext/System/data/apps/gcc/gcc-9.1.0/bin/gcc
COLLECT_LTO_WRAPPER=/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ./gcc-9.1.0/configure --prefix=/stornext/System/data/apps/gcc/gcc-9.1.0 --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,fortran,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --disable-multilib --build=x86_64-redhat-linux
Thread model: posix
gcc version 9.1.0 (GCC)
COMPILER_PATH=/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/:/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/:/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/
LIBRARY_PATH=/stornext/System/data/apps/gcc/gcc-9.1.0/lib64/../lib64/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/../lib64/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/stornext/System/data/tools/openSSL/openSSL-1.1.1n/lib/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib64/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/:/stornext/System/data/nvidia/cuda/cuda-11.4/lib64/:/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-std=c99' '-O3' '-D' 'NDEBUG' '-v' '-rdynamic' '-o' 'unit' '-std=c99' '-fPIC' '-Wextra' '-Wpedantic' '-Wshadow' '-O3' '-g' '-mtune=generic' '-march=x86-64'
/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/collect2 -plugin /stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/liblto_plugin.so -plugin-opt=/stornext/System/data/apps/gcc/gcc-9.1.0/libexec/gcc/x86_64-redhat-linux/9.1.0/lto-wrapper -plugin-opt=-fresolution=/vast/scratch/users/yang.e/tmp/cc5maLr7.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --eh-frame-hdr --hash-style=gnu -m elf_x86_64 -export-dynamic -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o unit /lib/../lib64/crt1.o /lib/../lib64/crti.o /stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/crtbegin.o -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib64/../lib64 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib/../lib64 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/stornext/System/data/tools/openSSL/openSSL-1.1.1n/lib -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib64 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib -L/stornext/System/data/nvidia/cuda/cuda-11.4/lib64 -L/stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/../../.. CMakeFiles/unit.dir/tests/unit.c.o libstreamvbyte_static.a -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /stornext/System/data/apps/gcc/gcc-9.1.0/lib/gcc/x86_64-redhat-linux/9.1.0/crtend.o /lib/../lib64/crtn.o
COLLECT_GCC_OPTIONS='-std=c99' '-O3' '-D' 'NDEBUG' '-v' '-rdynamic' '-o' 'unit' '-std=c99' '-fPIC' '-Wextra' '-Wpedantic' '-Wshadow' '-O3' '-g' '-mtune=generic' '-march=x86-64'
[100%] Built target unit
[ 15%] Performing install step for 'streamvbyte'
[ 33%] Built target streamvbyte_static
[ 66%] Built target streamvbyte
[ 75%] Built target example
[ 83%] Built target perf
[ 91%] Built target writeseq
[100%] Built target unit
Install the project...
-- Install configuration: "Release"
-- Installing: /home/users/allstaff/yang.e/vast-scratch/dorado-0.1.1-src-generic/cmake-build/streamvbyte_lib/include/streamvbyte.h
-- Installing: /home/users/allstaff/yang.e/vast-scratch/dorado-0.1.1-src-generic/cmake-build/streamvbyte_lib/include/streamvbytedelta.h
-- Installing: /home/users/allstaff/yang.e/vast-scratch/dorado-0.1.1-src-generic/cmake-build/streamvbyte_lib/include/streamvbyte_zigzag.h
-- Installing: /home/users/allstaff/yang.e/vast-scratch/dorado-0.1.1-src-generic/cmake-build/streamvbyte_lib/lib/libstreamvbyte.so
-- Installing: /home/users/allstaff/yang.e/vast-scratch/dorado-0.1.1-src-generic/cmake-build/streamvbyte_lib/lib/libstreamvbyte_static.a
[ 16%] Completed 'streamvbyte'
[ 16%] Built target streamvbyte
[ 17%] Building CXX object dorado/3rdparty/hdf_plugins/vbz/CMakeFiles/vbz.dir/v0/vbz_streamvbyte.cpp.o
[ 19%] Building CXX object dorado/3rdparty/hdf_plugins/vbz/CMakeFiles/vbz.dir/v1/vbz_streamvbyte.cpp.o
[ 20%] Building CXX object dorado/3rdparty/hdf_plugins/vbz/CMakeFiles/vbz.dir/vbz.cpp.o
[ 21%] Linking CXX static library ../../../../lib/libvbz.a
[ 21%] Built target vbz
[ 22%] Building CXX object dorado/3rdparty/hdf_plugins/vbz_plugin/CMakeFiles/vbz_hdf_plugin.dir/vbz_plugin.cpp.o
[ 23%] Linking CXX static library ../../../../lib/libvbz_hdf_plugin.a
[ 23%] Built target vbz_hdf_plugin
[ 24%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_crypt.c.o
[ 25%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_os.c.o
[ 26%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_strm.c.o
[ 28%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_strm_buf.c.o
[ 29%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_strm_mem.c.o
[ 30%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_strm_split.c.o
[ 31%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_zip.c.o
[ 32%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_zip_rw.c.o
[ 33%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_strm_zlib.c.o
[ 34%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_crypt_openssl.c.o
[ 35%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_os_posix.c.o
[ 37%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_strm_os_posix.c.o
[ 38%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_compat.c.o
[ 39%] Linking C static library libminizip.a
[ 39%] Built target minizip
[ 40%] Building CXX object dorado/3rdparty/elzip/CMakeFiles/elzip.dir/src/elzip.cpp.o
[ 41%] Building CXX object dorado/3rdparty/elzip/CMakeFiles/elzip.dir/src/unzipper.cpp.o
[ 42%] Building CXX object dorado/3rdparty/elzip/CMakeFiles/elzip.dir/src/zipper.cpp.o
[ 43%] Linking CXX static library libelzip.a
[ 43%] Built target elzip
[ 44%] Building CXX object dorado/3rdparty/edlib/CMakeFiles/edlib.dir/edlib/src/edlib.cpp.o
[ 46%] Linking CXX static library ../../../lib/libedlib.a
[ 46%] Built target edlib
[ 47%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/nn/CRFModel.cpp.o
[ 48%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/nn/RemoraModel.cpp.o
[ 49%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/read_pipeline/ReadPipeline.cpp.o
[ 50%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/read_pipeline/ScalerNode.cpp.o
[ 51%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/read_pipeline/StereoDuplexEncoderNode.cpp.o
[ 52%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/read_pipeline/BasecallerNode.cpp.o
[ 53%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/read_pipeline/ModBaseCallerNode.cpp.o
[ 55%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/read_pipeline/WriterNode.cpp.o
[ 56%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/read_pipeline/BaseSpaceDuplexCallerNode.cpp.o
[ 57%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/data_loader/DataLoader.cpp.o
[ 58%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/decode/beam_search.cpp.o
[ 59%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/decode/fast_hash.cpp.o
[ 60%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/decode/CPUDecoder.cpp.o
[ 61%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/modbase/remora_encoder.cpp.o
[ 62%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/modbase/remora_scaler.cpp.o
[ 64%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/modbase/remora_utils.cpp.o
[ 65%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/utils/base_mod_utils.cpp.o
[ 66%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/utils/compat_utils.cpp.o
[ 67%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/utils/log_utils.cpp.o
[ 68%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/utils/models.cpp.o
[ 69%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/utils/sequence_utils.cpp.o
[ 70%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/utils/stitch.cpp.o
[ 71%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/utils/tensor_utils.cpp.o
[ 73%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/utils/bam_utils.cpp.o
[ 74%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/utils/duplex_utils.cpp.o
[ 75%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/decode/GPUDecoder.cpp.o
[ 76%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/nn/CudaCRFModel.cpp.o
[ 77%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/utils/cuda_utils.cpp.o
[ 78%] Linking CXX static library lib/libdorado_lib.a
[ 78%] Built target dorado_lib
[ 79%] Building CXX object CMakeFiles/dorado.dir/dorado/main.cpp.o
[ 80%] Building CXX object CMakeFiles/dorado.dir/dorado/cli/duplex.cpp.o
[ 82%] Building CXX object CMakeFiles/dorado.dir/dorado/cli/basecaller.cpp.o
[ 83%] Building CXX object CMakeFiles/dorado.dir/dorado/cli/benchmark.cpp.o
[ 84%] Building CXX object CMakeFiles/dorado.dir/dorado/cli/download.cpp.o
[ 85%] Linking CXX executable bin/dorado
[ 85%] Built target dorado
[ 86%] Building CXX object dorado/3rdparty/hdf_plugins/vbz/test/CMakeFiles/vbz_test.dir/streamvbyte_test.cpp.o
[ 87%] Building CXX object dorado/3rdparty/hdf_plugins/vbz/test/CMakeFiles/vbz_test.dir/vbz_test.cpp.o
[ 88%] Building CXX object dorado/3rdparty/hdf_plugins/vbz/test/CMakeFiles/vbz_test.dir/main.cpp.o
[ 89%] Linking CXX executable ../../../../../bin/vbz_test
[ 89%] Built target vbz_test
[ 91%] Building CXX object tests/CMakeFiles/dorado_tests.dir/main.cpp.o
[ 92%] Building CXX object tests/CMakeFiles/dorado_tests.dir/Fast5DataLoaderTest.cpp.o
[ 93%] Building CXX object tests/CMakeFiles/dorado_tests.dir/TensorUtilsTest.cpp.o
[ 94%] Building CXX object tests/CMakeFiles/dorado_tests.dir/MathUtilsTest.cpp.o
[ 95%] Building CXX object tests/CMakeFiles/dorado_tests.dir/ReadTest.cpp.o
[ 96%] Building CXX object tests/CMakeFiles/dorado_tests.dir/RemoraEncoderTest.cpp.o
[ 97%] Building CXX object tests/CMakeFiles/dorado_tests.dir/SequenceUtilsTest.cpp.o
[ 98%] Building CXX object tests/CMakeFiles/dorado_tests.dir/StitchTest.cpp.o
[100%] Linking CXX executable dorado_tests
[100%] Built target dorado_tests
from dorado.
iiSeymour
commented on June 6, 2024
@edoyango we've upgraded to CUDA 11.7 in the latest release and I'd be keen to know if you still see any performance differences when compiling from source.
$ dorado -vv
dorado: 0.2.1+c70423e+cu117
libtorch: 1.13.1+cu117
from dorado.
edoyango
commented on June 6, 2024
hi @iiSeymour I've had a go and looks like the discrepancy is resolved:
# prebuilt binaries
[2023-02-20 12:30:40.471] [info] > Creating basecall pipeline
[2023-02-20 12:34:03.849] [info] > Reads basecalled: 20000
[2023-02-20 12:34:03.849] [info] > Samples/s: 3.687579e+06
[2023-02-20 12:34:03.849] [info] > Finished
# built from source
[2023-02-20 12:51:12.939] [info] > Creating basecall pipeline
[2023-02-20 12:54:42.682] [info] > Reads basecalled: 20000
[2023-02-20 12:54:42.682] [info] > Samples/s: 3.566496e+06
[2023-02-20 12:54:42.682] [info] > Finished
Thanks for keeping me up-to-date regarding this!
from dorado.
Related Issues (20)
- Basecaller with --modified-bases-models option HOT 2
- CUDA device requested but no devices found HOT 4
- Basecall process 100% but memory is still occupied for a long time HOT 4
- Empty values for columns "template_start" and "template_duration" in summary for barcoded reads HOT 1
- Dafult preset lr:hq for Dorado version-v0.6 HOT 1
- Problem building with cmake on an intel macbook pro 2020. HOT 5
- Running duplex still 0% for the last 12 hours HOT 8
- Barcode demultiplexing with dorado 0.6.1 HOT 2
- basecalling with hac v4.2 vs v4.3 model HOT 1
- Trimming existing datasets returns malformed fastq files HOT 1
- Corrupted aux data for read HOT 3
- Putting a link to Dorado binaries in a public workflow or document
- Read splitting HOT 3
- No POD5 or FAST5 data found in path HOT 3
- dorado v7.3.9 adapter trimming HOT 1
- trimming confusion HOT 10
- Dorado demultiplexing question HOT 2
- Dorado basecalling what's next ? Correlation with modkit output and transposable elements HOT 2
- automatic model selection, bps not matching with expected HOT 4
- Can't use multi-instance GPU (MIG) HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dorado.