ncbi / ncbi-vdb Goto Github PK

View Code? Open in Web Editor NEW

86.0 26.0 42.0 111.28 MB

ncbi-vdb

License: Other

Makefile 0.33% C++ 11.48% Shell 0.12% Python 0.92% Perl 0.43% C 85.74% Yacc 0.33% Lex 0.11% Roff 0.01% CMake 0.53%

ncbi-vdb's Introduction

The NCBI SRA ( Sequence Read Archive )

Contact:

email: [email protected]

Change Log

Please check the CHANGES.md file for change history.

NCBI VDB

The SRA Toolkit and SDK from NCBI is a collection of tools and libraries for using data in the INSDC Sequence Read Archives.

ANNOUNCEMENTS:

February 10, 2022

NCBI's SRA changed the source build system to use CMake in toolkit release 3.0.0. This change is an important step to improve developers' productivity as it provides unified cross platform access to support multiple build systems. This change affects developers building NCBI SRA tools from source. Old makefiles and build systems are longer be supported.

This change also includes the structure of GitHub repositories, which underwent consolidation to provide an easier environment for building tools and libraries (NGS libs and dependencies are consolidated). Consolidation of NGS libraries and dependencies provides better usage scope isolation and makes building more straightforward.

Affected repositories

ncbi/ngs

This repository is frozen. All future development will take place in GitHub repository ncbi/sra-tools, under subdirectory ngs/.

ncbi/ncbi-vdb (This repository)

This project's build system is based on CMake. The libraries providing access to SRA data in VDB format via the NGS API have moved to GitHub repository ncbi/sra-tools.

Old (base URL: https://github.com/ncbi/ncbi-vdb)	New (base URL: https://github.com/ncbi/sra-tools)
`libs/ngs`	`ngs/ncbi/ngs`
`libs/ngs-c++`	`ngs/ncbi/ngs-c++`
`libs/ngs-jni`	`ngs/ncbi/ngs-jni`
`libs/ngs-py`	`ngs/ncbi/ngs-py`
`libs/vdb-sqlite`	`libs/vdb-sqlite`
`test/ngs-java`	`test/ngs-java`
`test/ngs-python`	`test/ngs-python`

ncbi/sra-tools

This project's build system is based on CMake. The project acquired some new components, as listed in the table above.

Antivirus Warnings

Windows Defender and a few other anti-virus tools have recently been falsely warning about test-sra.exe.

We suggest verifying MD5 checksums after downloading.

For additional information on using, configuring, and building the toolkit, please visit our wiki or our web site at NCBI

SRA Toolkit Development Team

ncbi-vdb's People

Contributors

Stargazers

Watchers

ncbi-vdb's Issues

VDBManagerPathType() returns kptNotFound under stress

I have been bench marking the loading of SRA records using the VDB API to stream sequence data (no quality or other info) on AWS. Similar to the fasterq-dump strategy, I am attempting to read each SRA record in parallel, but using the Message Passing Interface (MPI) instead of just threads. Each MPI rank opens and reads a non-overlapping slice of an SRA record.

For a number of parallel MPI ranks gets larger than about 32, I've noticed that VDBManagerPathType() starts returning kptNotFound for about 10% of the MPI processes. I've been able to work around this by retrying the call to VDBManagerPathType() after waiting 5 seconds. Is there a good way to read an SRA record in parallel, ideally using 100's of independent, but concurrent, processes? I am interested in extracting reads from an SRA file as fast as AWS will allow.

I was assuming that the data is stored in an S3 bucket and that parallel access would be okay. I'm not exactly sure where the data is being stored, since the srapath command returns:
https://locate.ncbi.nlm.nih.gov/sdlr/sdlr.fcgi?jwt=<long string of characters removed>.

fatal error: 'ext/mbedtls/base64.h' file not found

Version 2.11.0 fails to compile:

/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/libs/cloud/aws-auth.c:34:10: fatal error: 'ext/mbedtls/base64.h' file not found
#include <ext/mbedtls/base64.h> /* vdb_mbedtls_base64_encode */
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.

File with this name is not present in the directory and there are no git submodules.

bitcpy

”bitcpy“ in data-buffer.c is undefine,how to solve this problem? Thank you!
Error reporting prompt：
/usr/local/sratool/ncbi-vdb-3.0.0/libs/klib/data-buffer.c:688: undefined reference to `bitcpy'

Enable Apple Silicon builds by adding `interfaces/cc/clang/aarch64`

In Homebrew, I was able to build v3.0.0 on Apple Silicon by adding an interfaces/cc/clang/aarch64 (Homebrew/homebrew-core#98654) linked from existing interfaces/cc/gcc/arm64

I will note that trying to build with GCC failed since ARCH=aarch64 rather than arm64, so build was looking for interfaces/cc/gcc/aarch64, which doesn't exist.

ncbi-vdb/build/env.cmake

Lines 68 to 71 in 34c6113

    
           elseif ( ${CMAKE_HOST_SYSTEM_PROCESSOR} STREQUAL "arm64") 
        
               set(ARCH "aarch64") 
        
           elseif ( ${CMAKE_HOST_SYSTEM_PROCESSOR} STREQUAL "aarch64") 
        
               set(ARCH "aarch64")

Static declaration of gettid follows non-static declaration

I have trouble installing ncbi-vdb through my package manager and from source, same error.

I've tried passing in CFLAGS that are recognized by the configure script, but dont pass through konfigure.pl properly.

The issue seems to center around libs/kproc/linux/sysmgr.c.

/home/matt/pckges/ncbi-vdb/libs/kproc/linux/sysmgr.c:36:7: error: static declaration of ‘gettid’ follows non-static declaration
   36 | pid_t gettid ( void )
      |       ^~~~~~
In file included from /usr/include/unistd.h:1170,
                 from /home/matt/pckges/ncbi-vdb/libs/kproc/linux/sysmgr.c:30:
/usr/include/bits/unistd_ext.h:34:16: note: previous declaration of ‘gettid’ was here
   34 | extern __pid_t gettid (void) __THROW;
      |                ^~~~~~
cc1: warning: unrecognized command line option ‘-Wno-c++11-extensions’
cc1: warning: unrecognized command line option ‘-Wno-c++98-compat-pedantic’
cc1: warning: unrecognized command line option ‘-Wno-c++98-compat’
make[3]: *** [/home/matt/pckges/ncbi-vdb/build/Makefile.rules:46: sysmgr.pic.o] Error 1
make[2]: *** [/home/matt/pckges/ncbi-vdb/build/Makefile.env:141: stdcompile] Error 2
make[1]: *** [Makefile:84: kproc] Error 2
make: *** [Makefile:47: libs] Error 2

When compiling SRA Tools & NCBI VDB in fedora36 environment, HDF5 static library or shared library file cannot be found

When I was configuring, the following problems occurred:
checking for fuse library... yes
checking for hdf5 library... no
checking for magic library... yes
checking for xml2 library... yes

In the konfigure.perl file, I see the following:
$library = '-Wl,-Bstatic -lhdf5 -Wl,-Bdynamic -ldl -lm -lz'
The explanation given is:build: recognize version oflibhdf5 that does not allow static linking and donot try to use it.

What I want to know is whether HDF5 will continue to be supported in version 3.0.0. If so, what should I do next?
After searching for relevant information, bioconda provides a solution:
.- setup/ konfigure.perl+++ setup/ konfigure.perl@-1654,7 +1654, 7 @r
$flags = $n;$log = "
int main( ) {
}ln'
}elsif (n eq " hdf5 ' ) {
$library = '-wl,-Bstatic -lhdf5 -wl,-Bdynamic -ldl -lm -lz ';$library = '-wl，-Bdynamic -lhdf5 -ldl -lm -lz';
$log = "#include <hdf5.h> in int main( ) { H5close
(; }n"
}elsif ($n eq 'fuse ' ) i
sflags = '-D_FILE_OFFSET_BITS=64' ;
--- tools/ pacbio-load/Makefile
+++ tools/pacbio-load/Makefilee-105,9 +105,9 @@
-stk-version :-skdf5 l
-sload -shdf5 v-lhdf5 v
-sncbi-wvdb l-sm l
-lm v
https://github.com/bioconda/bioconda-recipes/blob/master/recipes/sra-tools/0001-No-libhdf5-libxml2-static-link-sra-tools.patch
I hope to get an accurate plan, and I expect that the premise of using HDF5 is to respect the development direction of the community

modify the compiler

Hello, how can I modify the compiler.When I use: ./configure cc=clang cxx=clang++, gcc is not modified to clang

Is there a way to force the release of the VDB memory cache with C++ SDK?

As a follow on to a previous issue related to the memory/disk cache that is maintained by the VDB library: is there a way to free the VDB memory cache when it is no longer needed?

My application first streams SRA data and then performs follow on (and potentially memory intensive) processing steps. I was hoping that deleting the ngs::ReadCollection object would free any associated memory cache, but this does not appear to be the case.

Encountered problems with GCC

Hi，when I was compiling NCBI VDB, an error occurred that caused the compilation to fail:
checking whether gcc accepts -wno-array-bounds... no
checking whether g++ accepts -static-libstdc++... no
How should I solve this problem？

Benchmarking parallel SRA downloads using prefetch on AWS

Since the question of downloading SRA runs in "parallel" using many cloud instances to run prefetch, I though I would share the following "pseudo-benchmark" results.

The x-axis shows the number of single core, 25GB/s AWS instances (c5n.large) and the y-axis shows the fraction of the downloads that were successful. Data was collected on June 19, starting at 12 noon MT.

This is a "pseudo" benchmark because it does not attempt to download the exact same set of SRA accessions, but instead downloads different SRA accessions that have very similar file sizes (I am searching a list of ~250K SRA accessions that have been sorted in order of number of bases).

Built target Test_KNS_proxy-with-scheme_failure

I'm getting this below status once try to execute the $ make

Consolidate compiler generated dependencies of target Test_KNS_refresh-expired
[ 98%] Building CXX object test/kns/CMakeFiles/Test_KNS_refresh-expired.dir/HttpFixture.cpp.o
[ 98%] Building CXX object test/kns/CMakeFiles/Test_KNS_refresh-expired.dir/KStableHttpFile.cpp.o
[ 99%] Building CXX object test/kns/CMakeFiles/Test_KNS_refresh-expired.dir/test-refresh-expired-url.cpp.o
[ 99%] Linking CXX executable /home/drsisu/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/test-bin/Test_KNS_refresh-expired
[ 99%] Built target Test_KNS_refresh-expired
Consolidate compiler generated dependencies of target Test_KNS_timeouts
[ 99%] Building CXX object test/kns/CMakeFiles/Test_KNS_timeouts.dir/test-timeouts.cpp.o
[ 99%] Linking CXX executable /home/drsisu/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/test-bin/Test_KNS_timeouts
[ 99%] Built target Test_KNS_timeouts
Consolidate compiler generated dependencies of target test-kns-mutual-authentication
[ 99%] Building CXX object test/kns/CMakeFiles/test-kns-mutual-authentication.dir/test-mutual-authentication.cpp.o
[ 99%] Building CXX object test/kns/CMakeFiles/test-kns-mutual-authentication.dir/gosha.cpp.o
[ 99%] Linking CXX executable /home/drsisu/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/test-bin/test-kns-mutual-authentication
[ 99%] Built target test-kns-mutual-authentication
Consolidate compiler generated dependencies of target test-kns-proxy-with-env
[ 99%] Building CXX object test/kns/CMakeFiles/test-kns-proxy-with-env.dir/test-proxy-with-env.cpp.o
[100%] Linking CXX executable /home/drsisu/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/test-bin/test-kns-proxy-with-env
[100%] Built target test-kns-proxy-with-env
Consolidate compiler generated dependencies of target Test_KNS_proxy
[100%] Building CXX object test/kns/CMakeFiles/Test_KNS_proxy.dir/test-proxy.cpp.o
[100%] Linking CXX executable /home/drsisu/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/test-bin/Test_KNS_proxy
[100%] Built target Test_KNS_proxy
Consolidate compiler generated dependencies of target Test_KNS_proxy-with-scheme
[100%] Building CXX object test/kns/CMakeFiles/Test_KNS_proxy-with-scheme.dir/test-proxy-with-scheme.cpp.o
[100%] Linking CXX executable /home/drsisu/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/test-bin/Test_KNS_proxy-with-scheme
[100%] Built target Test_KNS_proxy-with-scheme
Consolidate compiler generated dependencies of target Test_KNS_proxy-with-scheme_failure
[100%] Building CXX object test/kns/CMakeFiles/Test_KNS_proxy-with-scheme_failure.dir/test-proxy-with-scheme_failure.cpp.o
[100%] Linking CXX executable /home/drsisu/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/test-bin/Test_KNS_proxy-with-scheme_failure
[100%] Built target Test_KNS_proxy-with-scheme_failure

how shall i proceed?

error: too few arguments to function ‘H5Oget_info_by_name3’

While following the instructions to compile from source, I faced this error after running ./configure && make:

Here is the colored version to help readability, the text of the error is also accessible in the collapsed section after the image:

The error log in text format

[ 57%] Building C object libs/ext/magic/CMakeFiles/magic.dir/apptype.c.o
[ 57%] Building C object libs/ext/magic/CMakeFiles/magic.dir/cdf.c.o
[ 57%] Building C object libs/ext/magic/CMakeFiles/magic.dir/cdf_time.c.o
[ 57%] Building C object libs/ext/magic/CMakeFiles/magic.dir/readcdf.c.o
[ 57%] Building C object libs/ext/magic/CMakeFiles/magic.dir/strlcat.c.o
[ 57%] Building C object libs/ext/magic/CMakeFiles/magic.dir/strlcpy.c.o
[ 57%] Linking C static library /home/mehrad/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/lib/libmagic.a
[ 57%] Built target magic
[ 57%] Building C object libs/hdf5/CMakeFiles/kdf5.dir/hdf5arrayfile.c.o
[ 57%] Building C object libs/hdf5/CMakeFiles/kdf5.dir/hdf5dir.c.o
In file included from /usr/include/H5public.h:32,
                 from /usr/include/hdf5.h:22,
                 from /home/mehrad/Applications/sra-tools/ncbi-vdb-3.0.0/libs/hdf5/hdf5dir.c:35:
/home/mehrad/Applications/sra-tools/ncbi-vdb-3.0.0/libs/hdf5/hdf5dir.c: In function ‘HDF5DirPathTypeOnBuffer’:
/home/mehrad/Applications/sra-tools/ncbi-vdb-3.0.0/libs/hdf5/hdf5dir.c:295:18: error: too few arguments to function ‘H5Oget_info_by_name3’
  295 |     herr_t h5e = H5Oget_info_by_name( self->hdf5_handle, buffer, &obj_info, H5P_DEFAULT );
      |                  ^~~~~~~~~~~~~~~~~~~
In file included from /usr/include/H5Apublic.h:22,
                 from /usr/include/hdf5.h:23:
/usr/include/H5Opublic.h:521:15: note: declared here
  521 | H5_DLL herr_t H5Oget_info_by_name3(hid_t loc_id, const char *name, H5O_info2_t *oinfo, unsigned fields,
      |               ^~~~~~~~~~~~~~~~~~~~
make[3]: *** [libs/hdf5/CMakeFiles/kdf5.dir/build.make:90: libs/hdf5/CMakeFiles/kdf5.dir/hdf5dir.c.o] Error 1
make[2]: *** [CMakeFiles/Makefile2:2063: libs/hdf5/CMakeFiles/kdf5.dir/all] Error 2
make[1]: *** [Makefile:146: all] Error 2
make: *** [/home/mehrad/Applications/sra-tools/ncbi-vdb-3.0.0/build/Makefile.env:106: cmake] Error 2

Some quick system information:

❯  gcc --version
gcc (GCC) 12.1.1 20220730
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


❯  inxi --system --width=80
System:
  Host: precisiontower Kernel: 5.15.59-1-MANJARO arch: x86_64 bits: 64
    Desktop: KDE Plasma v: 5.24.6 Distro: Manjaro Linux


❯  lsb_release -a
LSB Version:	n/a
Distributor ID:	ManjaroLinux
Description:	Manjaro Linux
Release:	21.3.6
Codename:	Ruah

Please let me know if you need further info.

/kns/http-file.c:1275:19: warning: this 'else' clause does not guard... [-Wmisleading-indentation]

Please kindly consider to give this warning some extra review and donate a few {}s for extra clarity.
Cheers,
Steffen

/build/ncbi-vdb-2.10.6+dfsg/libs/kns/http-file.c: In function 'KHttpFileMake':
/build/ncbi-vdb-2.10.6+dfsg/libs/kns/http-file.c:1275:19: warning: this 'else' clause does not guard... [-Wmisleading-indentation]
 1275 |                   else
      |                   ^~~~
/build/ncbi-vdb-2.10.6+dfsg/libs/kns/http-file.c:1277:21: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the 'else'
 1277 |                     if ( rc == 0 )
      |                     ^~

libs/search requires -msse2 switch

Hi,

I received a bug report for the Debian package and this patch is one option to solve the issue.

Kind regards

      Andreas.

Memory leak in ngs::ReadCollection and/or ncbi::NGS::openReadCollection when streaming SRA records?

There appears to be a memory leak in ngs::ReadCollection and/or ncbi::NGS::openReadCollection when streaming SRA records (as opposed to reading SRA records that have been previously downloaded using the prefetch utility). This leak can be observed using the following code example:

#include <iostream>
#include <sstream>
#include <sys/types.h> // For getpid()
#include <unistd.h>
#include <ncbi-vdb/NGS.hpp> // For openReadCollection

using namespace std;

int main(int argc, char *argv[])
{
        const string accession = "SRR360112";

        cout << "Starting memory test for " << accession << endl;

        const pid_t pid = getpid();

        for(size_t iteration = 0;iteration < 100;++iteration){

                stringstream mem_check;
                // Very simple way to monitor memory usage on Linux
                mem_check << "cat /proc/" << pid << "/status | grep VmSize";

                system( mem_check.str().c_str() );

                // Open the SRA accession, but don't load any data ...
                ngs::ReadCollection run(  ncbi::NGS::openReadCollection(accession) );
        }

        return EXIT_SUCCESS;
}

Running this simple code on an AWS instance that only opens the SRA record, and does not actually load any data, yields the following memory consumption:

The growing memory consumption is observed for many SRA records, not just the specific example of SRR360112 that is used above.

If the SRA record is not already stored locally (i.e. by first running the prefetch utility), the memory consumption grows after every iteration. This growing memory consumption becomes a problem when loading data from thousands of SRA records (especially when using AWS EC2 instances for which extra RAM means extra $$$).

Some builtins that vdb uses are gcc-specific and don't compile with clang.

For example, __builtin_ia32_shufps and __builtin_ia32_shufpd are gcc-specific.

You should replace them with primitives, _mm_shuffle_ps and _mm_shuffle_pd, or better yet, you can use the SIMD library instead, like libsimdpp: https://github.com/p12tic/libsimdpp

VCursorCellDataDirect returns a zero length read ... but shouldn't?

I am using VCursorCellDataDirect to stream sequence data from SRA files into my application. In general, this works fine. However, I have run across an SRA record for which the API call:
VCursorCellDataDirect(cur, index, read_col_id, &elem_bits, (const void **)&str.addr, &boff, &str.len ) successfully returns, and sets the str.len value to zero.

A subsequent call to VCursorCellDataDirect( cur, index, read_len_col_id, &elem_bits, (const void **)&read_len_array, &boff, &num_read_len ) successfully returns with num_read_len equal to 2 and read_len_array[0]=74 and read_len_array[1]=74.

Here is my question -- if the returned string length is zero in the first call to VCursorCellDataDirect, is it an error to attempt to load the read length array for the same read index? I'm not sure why the sum of the read length array elements aren't zero (for consistency). The SRA record in question is SRR1614306.

error: pshufb is not supported on generic64.aes

Hi, I compiled the recent git version of ncbi-vdb, including ver 2.8.2, in my arch linux server (updated recently), it gave this error when testing after make. how could i fix this. thx in advance.

gcc -c -DNDEBUG -m64 -DLINUX -DUNIX -D_GNU_SOURCE -D_REENTRANT -D_FILE_OFFSET_BITS=64 -DPKGNAME=linux64 -D_ARCH_BITS=_SIZEOF_POINTER_*CHAR_BIT -DLIBPREFIX=lib
-DSHLIBEXT=so -I/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/interfaces/override -I/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/libs/krypto/linux -I/home/
eastasia/softhouse/biotools/git_svn/ncbi-vdb/libs/krypto/unix -I/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/libs/krypto -I/home/eastasia/softhouse/biotools/git_s
vn/ncbi-vdb/interfaces -I/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/interfaces/cc/gcc/x86_64 -I/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/interfaces/cc/
gcc -I/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/interfaces/os/linux -I/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/interfaces/os/unix -I/home/eastasia/so
fthouse/biotools/git_svn/ncbi-vdb/interfaces/ext -I/usr/include -I. -MD -o aes-ncbi.aes-ni.pic.o -fPIC -O3 -Wno-variadic-macros -fno-strict-aliasing -U_FORTIFY_SOURCE -
D_FORTIFY_SOURCE=1 -Wall -D_LIBRARY -DUSEVEC -DUSEVECREG -DUSEAESNI -funsafe-math-optimizations -mmmx -msse -msse2 -msse3 -mssse3 -msse4.1 -maes -Wa,-march=generic64+ae
s -Wa,-ahlms=/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/libs/krypto/aes-ncbi.aes-ni.pic.o.list /home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/libs/krypto/ae
s-ncbi.c
In file included from /home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/libs/krypto/aes-ncbi.c:29:0:
/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/libs/krypto/ncbi-priv.h:60:33: warning: ‘KAESBlockCipherVecAesNiaes_ncbi_name’ defined but not used [-Wunused-const-v
ariable=]
# define AESBCMEMBER(name) KAESBlockCipherVecAesNi##name
^
/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/libs/krypto/aes-ncbi.c:2292:19: note: in expansion of macro ‘AESBCMEMBER’
static const char AESBCMEMBER(aes_ncbi_name) [] = CLASS_STRING(AES_CLASS);
^~~~~~~~~~~
/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/libs/krypto/aes-ncbi.c:95:11: warning: ‘vec_10’ defined but not used [-Wunused-const-variable=]
CipherVec vec_10 = { 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, 0x10,
^~~~~~
/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/libs/krypto/aes-ncbi.c: In function ‘KAESBlockCipherVecAesNiMake’:
/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/libs/krypto/aes-ncbi.c:2507:34: warning: ‘d’ may be used uninitialized in this function [-Wmaybe-uninitialized]
uint32_t level = 1, a, b, c, d;
^
/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/libs/krypto/aes-ncbi.c:2507:31: warning: ‘c’ may be used uninitialized in this function [-Wmaybe-uninitialized]
uint32_t level = 1, a, b, c, d;
^
/tmp/cce3Rlg0.s: Assembler messages:
/tmp/cce3Rlg0.s:70: Error: pshufb' is not supported on generic64.aes'
..
..
...
/tmp/cce3Rlg0.s:611: Error: pshufb' is not supported on generic64.aes'
/tmp/cce3Rlg0.s:615: Error: pshufb' is not supported on generic64.aes'
make[8]: *** [/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/libs/krypto/Makefile:194: aes-ncbi.aes-ni.pic.o] Error 1
make[8]: Target '/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/ncbi-out/ncbi-vdb/linux/gcc/x86_64/rel/std' not remade because of errors.
reattempting without AES-NI instructions
Installing libraries to /home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/test/install/root/lib64
Installing includes to /home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/test/install/root/include
Installing configuration files to /home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/test/install/root/etc/ncbi
Updating /home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/test/install/root/usr/include/ncbi-vdb
Updating /home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/test/install/root/etc/profile.d/ncbi-vdb.[c]sh
Use $NCBI_VDB_LIBDIR in your link commands, e.g.:
ld -L$NCBI_VDB_LIBDIR -lncbi-vdb ...
Please add /home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/test/install/root/lib64 to your LD_LIBRARY_PATH, e.g.:
export LD_LIBRARY_PATH=/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/test/install/root/lib64:$LD_LIBRARY_PATH
Use /home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/test/install/root/lib64 in your link commands, e.g.:
export NCBI_VDB_LIBDIR=/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/test/install/root/lib64
ld -L$NCBI_VDB_LIBDIR -lncbi-vdb ...
make[3]: Leaving directory '/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/test/install'
make[2]: Leaving directory '/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/test/install'
make[1]: Target 'by_default' not remade because of errors.
make[1]: Leaving directory '/home/eastasia/softhouse/biotools/git_svn/ncbi-vdb/test'

ktst test failure

Getting a fatal error in test-ktst:

OS: RHEL 8
Compiler: GCC 9.2.0

Run /home/myname/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/test-bin/test-ktst
Running 12 test cases...
Parameter (0 s)
Requires (0 s)
ParentProcess (0 s)
ChildProcessOk (0 s)
/ifs/opt/src/ncbi/ncbi-vdb-2.10.9/test/ktst/ktsttest.cpp(123): fatal error in fixture: ChildProcessFail: failure in a child process
ChildProcessFail aborted
ChildProcessFail (0 s)
fatal error: signal: SIGFPE(Floating point exception)
/ifs/opt/src/ncbi/ncbi-vdb-2.10.9/test/ktst/ktsttest.cpp(129): last checkpoint
Test is aborted
ChildProcessAbort (0 s)
child process timed out
ChildProcessTimeout (1 s)
ArgHandlerCalled (0 s)
SharedSucceed (0 s)
/ifs/opt/src/ncbi/ncbi-vdb-2.10.9/test/ktst/ktsttest.cpp:156: error in "SharedFailed.Shared": check ok failed
SharedFailed (0 s)
/ifs/opt/src/ncbi/ncbi-vdb-2.10.9/test/ktst/ktsttest.cpp:156: error in "SharedFailedAndDetected.Shared": check ok failed
SharedFailedAndDetected (0 s)
SharedSucceedInBlock (0 s)

make install error: KFG lib not compiling

Hey NCBI;

I have a bit of a wonky system (CentOS5) for scientific computing and can't run the 2.8.0 version of sratools due to GLIBC2.7 being absent. I can compile ngs-sdk and I'm trying to build ncbi-vdb on this system but I get an error:

$ git clone https://github.com/ncbi/ncbi-vdb.git
... [works]
$ ncbi-vdb/configure --prefix=/home/ababaian/software/ncbi/ncbi-vdb/
...[works]
$ make default install -C ncbi-vdb/
...
make[3]: Entering directory `/home/ababaian/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/obj/libs/kns'
make[3]: Nothing to be done for `/home/ababaian/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/std'.
make[3]: Leaving directory `/home/ababaian/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/obj/libs/kns'
make[2]: Leaving directory `/home/ababaian/software/ncbi/ncbi-vdb/libs/kns'
make[2]: Entering directory `/home/ababaian/software/ncbi/ncbi-vdb/libs/kfg'
make[3]: Entering directory `/home/ababaian/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/obj/libs/kfg'
gcc -c   -DNDEBUG -m64   -DLINUX -DUNIX -D_GNU_SOURCE -D_REENTRANT -D_FILE_OFFSET_BITS=64 -DPKGNAME=linux64 -D_ARCH_BITS=__SIZEOF_POINTER__*__CHAR_BIT__ -DLIBPREFIX=lib -DSHLIBEXT=so  -I/home/ababaian/software/ncbi/ncbi-vdb/interfaces/override -I/home/ababaian/software/ncbi/ncbi-vdb/libs/kfg/linux -I/home/ababaian/software/ncbi/ncbi-vdb/libs/kfg/unix -I/home/ababaian/software/ncbi/ncbi-vdb/libs/kfg -I/home/ababaian/software/ncbi/ncbi-vdb/interfaces -I/home/ababaian/software/ncbi/ncbi-vdb/interfaces/cc/gcc/x86_64 -I/home/ababaian/software/ncbi/ncbi-vdb/interfaces/cc/gcc -I/home/ababaian/software/ncbi/ncbi-vdb/interfaces/os/linux -I/home/ababaian/software/ncbi/ncbi-vdb/interfaces/os/unix -I/home/ababaian/software/ncbi/ncbi-vdb/interfaces/ext -I/home/ababaian/software/ncbi/ngs/ngs-sdk -I. -MD -o config.pic.o -fPIC -O3 -Wno-variadic-macros -fno-strict-aliasing -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -Wall -D_LIBRARY /home/ababaian/software/ncbi/ncbi-vdb/libs/kfg/config.c -D__mod_name__=libs/kfg -D__file_name__=config -D__file_ext__=c
/home/ababaian/software/ncbi/ncbi-vdb/libs/kfg/config.c: In function ‘add_predefined_nodes’:
/home/ababaian/software/ncbi/ncbi-vdb/libs/kfg/config.c:2772: error: ‘__SIZEOF_POINTER__’ undeclared (first use in this function)
/home/ababaian/software/ncbi/ncbi-vdb/libs/kfg/config.c:2772: error: (Each undeclared identifier is reported only once
/home/ababaian/software/ncbi/ncbi-vdb/libs/kfg/config.c:2772: error: for each function it appears in.)
make[3]: *** [config.pic.o] Error 1
make[3]: Leaving directory `/home/ababaian/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/obj/libs/kfg'
make[2]: *** [stdcompile] Error 2
make[2]: Leaving directory `/home/ababaian/software/ncbi/ncbi-vdb/libs/kfg'
make[1]: *** [kfg] Error 2
make[1]: Leaving directory `/home/ababaian/software/ncbi/ncbi-vdb/libs'
make: *** [libs] Error 2
make: Leaving directory `/home/ababaian/software/ncbi/ncbi-vdb'

Any ideas how this can be solved?

Change to 2.8.1 release tag / checksum error?

Hello, I was wondering if there were any changes that caused version 2.8.1 to be retagged? There was a checksum error in a Homebrew package for sratoolkit and they were wondering if it was due to a release tag change and wanted to see if it could be confirmed https://github.com/Homebrew/homebrew-science/pull/4767

Reference Workflow for Fastq->SRA File (VDB Conversion)

Hello,
Can anyone point me to a reference to the code that is used by the SRA project to convert from a user's submitted FASTQ files to the VDB/.sra format that is stored then in the SRA? I know the format is not fully documented, but understanding this part of the ETL pipeline would be really helpful in understanding this format. Or any example code/pipeline would be helpful.
Thank you.

The gethostbyname() and gethostbyaddr() functions are deprecated as among other things they are not IPv6 ready

Applications should use getaddrinfo(3) and getnameinfo(3) instead. See also https://udrepper.livejournal.com/16116.html

Uneven reading time within a SRA file

I am downloading SRA files with prefetch and then reading them in C++ by iterating over ngs::ReadCollection. All calculations are running in the Cloud on AWS. I have found a small number of seeingly "pathological" SRA runs that take longer to read as the iteration progresses through the file.

For example, the graph below shows the time required to read sequential, 0.1% chunks of ERR3212419 (where the x-axis is the cumulative number of reads read as a percentage of the total number of reads in the SRA run). As shown in the graph, the first 16% of reads can be read from disk relatively quickly (approximately 2 seconds per 0.1% chunk). However, the time to read the same number of reads then jumps to approximately 12 seconds, and then jumps again to over 100 seconds. (I stopped after loading 21% of the reads).

Is there a way to read this SRA record (and records like it), so that the time required to read different parts of the file is even? This is important because I would like to read SRA records (from disk) in parallel, and the uneven time-to-read makes for significant load imbalances. In this example, parallel workers reading near the beginning of the file finish much faster than the parallel worker reading near the end of the file.

Parallel use of prefetch of SRA files from AWS S3 leads to NCBI abuse message & blocking?

Hi all,

My apologies if this is not the right place to file this issue.

I'm downloading SRA files using prefetch on an AWS EC2 instance in the
us-east-1 region. I've used vdb-config -i to make sure that I'm using
AWS S3 to fetch the SRA files. Here's the output of srapath to show that I'm fetching from S3:

> srapath ERR169824
https://sra-pub-run-5.s3.amazonaws.com/ERR169824/ERR169824.1

I'm on a beefy EC2 instance with 96 cores, with each core running prefetch and then processing the SRA file with blast_vdb. Sometimes the two processes complete in a few seconds, sometimes it takes them up to 4 minutes. After my job was running for a while, I got this message:

Critical: (307.10) [NCBI-MESSAGE]  [DISPD#ID2@sutils201] Auto-blocked
for abuse, please contact NCBI help desk <[email protected]>

Are there resource limitations when downloading on S3, as there are with
using EUtils? Time-of-day, time-of-week, and/or rate limits? How to avoid this issue?

Thank you!

-bash: vdb-config: command not found

Hello. After cd into bin, and trying to execute vdb-config, vdb-config -i --interactive-mode textual, and vdb-config --restore-defaults, with and without ./, and ./[command].sh, I am always receiving the following errors:

-bash: vdb-config: command not found
or
-bash: ./vdb-config: cannot execute binary file: Exec format error

How do I overcome this error?
This is after downloading the sratoolkit.3.0.0-mac64.

Error: `pshufb' is not supported on `generic64.aes'

I got the this assembler error. I have modern Haswell CPU.

016-08-21 11:20:37 +1000

make
install

Checking make status of object libraries...
gcc -c   -DNDEBUG -m64   -DLINUX -DUNIX -D_GNU_SOURCE -D_REENTRANT -D_FILE_OFFSET_BITS=64 -DPKGNAME=linux64 -D_ARCH_BITS
=__SIZEOF_POINTER__*__CHAR_BIT__ -DLIBPREFIX=lib -DSHLIBEXT=so  -I/tmp/sratoolkit--ncbi-vdb-20160821-45016-1l3jwn7/ncbi-
vdb-2.7.0/interfaces/override -I/tmp/sratoolkit--ncbi-vdb-20160821-45016-1l3jwn7/ncbi-vdb-2.7.0/libs/krypto/linux -I/tmp
/sratoolkit--ncbi-vdb-20160821-45016-1l3jwn7/ncbi-vdb-2.7.0/libs/krypto/unix -I/tmp/sratoolkit--ncbi-vdb-20160821-45016-
1l3jwn7/ncbi-vdb-2.7.0/libs/krypto -I/tmp/sratoolkit--ncbi-vdb-20160821-45016-1l3jwn7/ncbi-vdb-2.7.0/interfaces -I/tmp/s
ratoolkit--ncbi-vdb-20160821-45016-1l3jwn7/ncbi-vdb-2.7.0/interfaces/cc/gcc/x86_64 -I/tmp/sratoolkit--ncbi-vdb-20160821-
45016-1l3jwn7/ncbi-vdb-2.7.0/interfaces/cc/gcc -I/tmp/sratoolkit--ncbi-vdb-20160821-45016-1l3jwn7/ncbi-vdb-2.7.0/interfa
ces/os/linux -I/tmp/sratoolkit--ncbi-vdb-20160821-45016-1l3jwn7/ncbi-vdb-2.7.0/interfaces/os/unix -I/tmp/sratoolkit--ncb
i-vdb-20160821-45016-1l3jwn7/ncbi-vdb-2.7.0/interfaces/ext -I/home/linuxbrew/.linuxbrew/Cellar/sratoolkit/2.7.0/include
-I. -MD -o aes-ncbi.aes-ni.pic.o -fPIC -O3 -Wno-variadic-macros -fno-strict-aliasing -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE
=1 -Wall -D_LIBRARY -DUSEVEC -DUSEVECREG -DUSEAESNI -funsafe-math-optimizations -mmmx -msse -msse2 -msse3 -mssse3 -msse4
.1 -maes -Wa,-march=generic64+aes -Wa,-ahlms=/tmp/sratoolkit--ncbi-vdb-20160821-45016-1l3jwn7/ncbi-vdb-2.7.0/libs/krypto
/aes-ncbi.aes-ni.pic.o.list /tmp/sratoolkit--ncbi-vdb-20160821-45016-1l3jwn7/ncbi-vdb-2.7.0/libs/krypto/aes-ncbi.c
/tmp/sratoolkit--ncbi-vdb-20160821-45016-1l3jwn7/ncbi-vdb-2.7.0/libs/krypto/aes-ncbi.c: In function 'KAESBlockCipherVecA
esNiMake':
/tmp/sratoolkit--ncbi-vdb-20160821-45016-1l3jwn7/ncbi-vdb-2.7.0/libs/krypto/aes-ncbi.c:2507:34: warning: 'd' may be used
 uninitialized in this function [-Wmaybe-uninitialized]
     uint32_t level = 1, a, b, c, d;
/tmp/sratoolkit--ncbi-vdb-20160821-45016-1l3jwn7/ncbi-vdb-2.7.0/libs/krypto/aes-ncbi.c:2507:31: warning: 'c' may be used
 uninitialized in this function [-Wmaybe-uninitialized]
     uint32_t level = 1, a, b, c, d;
                               ^
/tmp/cc104hCB.s: Assembler messages:
/tmp/cc104hCB.s:95: Error: `pshufb' is not supported on `generic64.aes'
/tmp/cc104hCB.s:99: Error: `pshufb' is not supported on `generic64.aes'
/tmp/cc104hCB.s:103: Error: `pshufb' is not supported on `generic64.aes'
/tmp/cc104hCB.s:111: Error: `pshufb' is not supported on `generic64.aes'
/tmp/cc104hCB.s:114: Error: `pshufb' is not supported on `generic64.aes'
/tmp/cc104hCB.s:117: Error: `pshufb' is not supported on `generic64.aes'
/tmp/cc104hCB.s:125: Error: `pshufb' is not supported on `generic64.aes'
/tmp/cc104hCB.s:128: Error: `pshufb' is not supported on `generic64.aes'
/tmp/cc104hCB.s:131: Error: `pshufb' is not supported on `generic64.aes'

make[4]: *** [aes-ncbi.aes-ni.pic.o] Error 1
reattempting without AES-NI instructions
Installing libraries to /home/linuxbrew/.linuxbrew/Cellar/sratoolkit/2.7.0/lib64
installing 'libncbi-vdb.a.2.7.0'... success
installing 'libncbi-vdb.so.2.7.0'... success
installing 'libncbi-wvdb.a.2.7.0'... success
installing 'libncbi-wvdb.so.2.7.0'... success
installing 'libncbi-ngs-c++.a.2.7.0'... success
Installing includes to /home/linuxbrew/.linuxbrew/Cellar/sratoolkit/2.7.0/include
Installing configuration files to /home/linuxbrew/.linuxbrew/Cellar/sratoolkit/2.7.0/lib64/ncbi/
Please add /home/linuxbrew/.linuxbrew/Cellar/sratoolkit/2.7.0/lib64 to your LD_LIBRARY_PATH, e.g.:
      export LD_LIBRARY_PATH=/home/linuxbrew/.linuxbrew/Cellar/sratoolkit/2.7.0/lib64:$LD_LIBRARY_PATH
Use /home/linuxbrew/.linuxbrew/Cellar/sratoolkit/2.7.0/lib64 in your link commands, e.g.:
      export NCBI_VDB_LIBDIR=/home/linuxbrew/.linuxbrew/Cellar/sratoolkit/2.7.0/lib64
      ld -L$NCBI_VDB_LIBDIR -lncbi-vdb ...

My CPU flags:

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc

Build from source errors

I can't seem to get ncbi-vdb to build. Am I doing something wrong?

git clone https://github.com/ncbi/ngs.git 
git clone https://github.com/ncbi/ncbi-vdb.git
cd ngs
./configure --prefix=install-ubuntu
make -j8

cd ../ncbi-vdb
./configure --prefix=install-ubuntu
make -j8

produces:

make[1]: Entering directory '/home/ciaran_ubuntu/tmp/tmp/ncbi-vdb/libs'
make[2]: Entering directory '/home/ciaran_ubuntu/tmp/tmp/ncbi-vdb/libs/ext'
make[3]: Entering directory '/home/ciaran_ubuntu/tmp/tmp/ncbi-vdb/libs/ext/zlib'
make[4]: Entering directory '/home/ciaran_ubuntu/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/obj/libs/ext/zlib'
/bin/sh: 1: /home/ciaran_ubuntu/tmp/tmp/ncbi-vdb/build/cc.sh: not found
make[4]: *** [/home/ciaran_ubuntu/tmp/tmp/ncbi-vdb/build/Makefile.rules:46: adler32.pic.o] Error 127
make[4]: Leaving directory '/home/ciaran_ubuntu/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/obj/libs/ext/zlib'
make[3]: *** [/home/ciaran_ubuntu/tmp/tmp/ncbi-vdb/build/Makefile.env:141: stdcompile] Error 2
make[3]: Leaving directory '/home/ciaran_ubuntu/tmp/tmp/ncbi-vdb/libs/ext/zlib'
make[2]: *** [Makefile:45: zlib] Error 2
make[2]: Leaving directory '/home/ciaran_ubuntu/tmp/tmp/ncbi-vdb/libs/ext'
make[1]: *** [Makefile:84: ext] Error 2
make[1]: Leaving directory '/home/ciaran_ubuntu/tmp/tmp/ncbi-vdb/libs'
make: *** [Makefile:47: libs] Error 2

Configuring with --with-ngs-sdk-prefix also results in the same output. Do you have any suggestions? Thanks

konfigure.perl versioned install path

https://github.com/ncbi/ncbi-vdb/blob/master/setup/konfigure.perl#L930

The current install behavior is incompatible with install prefix containing the program's version number:

...
make[1]: Leaving directory `/mnt/galaxy/tools/sra_toolkit/2.4.5/matt-shirley/ncbi_sra_toolkit/ef71f5982df5/src/ncbi-vdb/libs'
NOTE - internal library libkff cannot be built: It requires 'libmagic' and its development headers.
NOTE - library libkdf5 cannot be built: It requires 'libhdf5' and its development headers.
installing 'libncbi-vdb.a.2.4.5'... ln: failed to create symbolic link `/mnt/galaxy/tools/sra_toolkit/2/matt-shirley/ncbi_sra_toolkit/ef71f5982df5/lib64/libncbi-vdb.a.2': No such file or directory
success
installing 'libncbi-vdb.so.2.4.5'... ln: failed to create symbolic link `/mnt/galaxy/tools/sra_toolkit/2/matt-shirley/ncbi_sra_toolkit/ef71f5982df5/lib64/libncbi-vdb.so.2': No such file or directory
success
installing 'libncbi-wvdb.a.2.4.5'... ln: failed to create symbolic link `/mnt/galaxy/tools/sra_toolkit/2/matt-shirley/ncbi_sra_toolkit/ef71f5982df5/lib64/libncbi-wvdb.a.2': No such file or directory
success
installing 'libncbi-wvdb.so.2.4.5'... ln: failed to create symbolic link `/mnt/galaxy/tools/sra_toolkit/2/matt-shirley/ncbi_sra_toolkit/ef71f5982df5/lib64/libncbi-wvdb.so.2': No such file or directory
success
installing 'libncbi-ngs-c++.a.2.4.5'... ln: failed to create symbolic link `/mnt/galaxy/tools/sra_toolkit/2/matt-shirley/ncbi_sra_toolkit/ef71f5982df5/lib64/libncbi-ngs-c++.a.2': No such file or directory
success
Installing includes to /mnt/galaxy/tools/sra_toolkit/2.4.5/matt-shirley/ncbi_sra_toolkit/ef71f5982df5/include
...

"prefetch -t http ..." produces empty files

I'm having trouble using "prefetch": it always creates 0-sized files in ncbi/public/sra.

It does work when using the aspera client, and the file it tries to download from NCBI can be obtained with wget correctly. Since the only "unusual" thing I can see is a transparent proxy between ncbi and our server, my best guess is that the issue is caused by something within kns. Perhaps KHttpFile() does not work nicely with the proxy. See below for the outout of strace -s1000 -ff prefetch -t http SRR444041. For some reason, right after the first chunk of data (4344 bytes) is received on fd 3 (with recvfrom()) from ncbi's webserver, the output file on fd 5 is closed without ever writing to it. After closing the output file, before shutdown() is called on fd 3, a few more calls to recvfrom() result in chunks of 1024 bytes each.

write(1, "2016-03-04T01:41:04 prefetch.2.5.7:  Downloading via http...\n", 61) = 61

open("/home/data/ncbi/public/sra/SRR444041.sra.tmp.93082.tmp", O_WRONLY|O_CREAT|O_TRUNC, 0664) = 5

poll([{fd=3, events=POLLOUT|POLLWRNORM|POLLWRBAND}], 1, 0) = 1 ([{fd=3, revents=POLLOUT|POLLWRNORM}])

sendto(3, "GET /srapub/SRR444041 HTTP/1.1\r\nHost: sra-download.ncbi.nlm.nih.gov\r\nAccept: 
*/*\r\nRange: bytes=0-1048575\r\nUser-Agent: linux64 sra-toolkit prefetch.2.5.7\r\n\r\n", 156, 0, NULL,
 0) = 156

poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND|0x2000}], 1, 0) = 0 (Timeout)

poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND|0x2000}], 1, 0) = 0 (Timeout)

poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND|0x2000}], 1, 30000) = 1 ([{fd=3, 
revents=POLLIN|POLLRDNORM}])

recvfrom(3, "HTTP/1.1 200 OK\r\nDate: Fri, 04 Mar 2016 01:41:04 GMT\r\nServer: Apache\r\nAccept-
Ranges: bytes\r\nCache-Control: max-age=31536000\r\nExpires: Sat, 04 Mar 2017 01:41:04 
GMT\r\nKeep-Alive: timeout=5, max=99999\r\nContent-Length: 4862109\r\nContent-Type: 
application/octet-stream\r\nLast-Modified: Sun, 12 Aug 2012 01:49:16 GMT\r\nVia: 1.1 
wsa2-man.ucdenver.pvt:80 (Cisco-WSA/8.8.0-085)\r\nConnection: keep-alive\r\n\r\nNCBI.sra\210\31
\3\5\1\0\0\0\274\6\0\0\0\0\0\0\5\0\0\0\221\6\0\0\0\0\272\5 \0063\6o\6\3\0col\17J^O\0\0\0\0h\1\0\0\1\5
\0\0\0\226\5\0\0\0\0!\1B\2`\3{\4\7\0ALTREAD\17J^O\0\0\0\0h\1\0\0\1\7\0\0\0\374\0\0"..., 65536, 0, 
NULL, NULL) = 4344

close(5)                                = 0

close(4)                                = 0

rename("/home/data/ncbi/public/sra/SRR444041.sra.tmp.93082.tmp", "/home/data/ncbi/public
/sra/SRR444041.sra") = 0

lstat("/home/data/ncbi/public/sra/SRR444041.sra.cache", 0x7ffebf99c120) = -1 ENOENT (No such file
 or directory)

lstat("/home/data/ncbi/public/sra/SRR444041.sra.tmp.93082.tmp", 0x7ffebf99b090) = -1 ENOENT (No 
such file or directory)

openat(AT_FDCWD, "/home/data/ncbi/public/sra/", 
O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4

getdents(4, /* 6 entries */, 32768)     = 208

getdents(4, /* 0 entries */, 32768)     = 0

close(4)                                = 0

lstat("/home/data/ncbi/public/sra/SRR444041.sra.lock", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

unlink("/home/data/ncbi/public/sra/SRR444041.sra.lock") = 0

write(1, "2016-03-04T01:41:04 prefetch.2.5.7: 1) 'SRR444041' was downloaded successfully\n", 79) = 
79

lstat("/home/data/ncbi/public/sra/SRR444041.sra", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

lstat("/home/data/ncbi/public/sra/SRR444041.sra", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

lstat("/home/data/ncbi/public/sra/SRR444041.sra", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

open("/home/data/ncbi/public/sra/SRR444041.sra", O_RDONLY) = 4

fstat(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

pread(4, "", 4096, 0)                   = 0

fstat(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

pread(4, "", 4096, 0)                   = 0

fstat(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

close(4)                                = 0

lstat("/home/data/ncbi/public/sra/SRR444041.sra", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

lstat("/home/data/ncbi/public/sra/SRR444041.sra", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

open("/home/data/ncbi/public/sra/SRR444041.sra", O_RDONLY) = 4

fstat(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

pread(4, "", 24, 0)                     = 0

close(4)                                = 0

shutdown(3, 1 /* send */)               = 0

recvfrom(3, "998ecf8427e *idx0\n727b8f00c024d9a650068e198b0f77eb *idx2\ndb29c9d245ee82c9d997020fc2c41ceb *data\n\0\200\n\2\0\0\1\0\201\200\200\34\0\0\4$\214\341\215r\334\310\265v\335F0\274\255\237n\211\"UD\241\200\314\\s>\367\177\221_\2U\224\250\266\375\2363\306\277\376~[\257\353\341\372v}\373\376\366\364_\374\353\357\327\267\313\333\333\317\237\227\313\353\217\247\377\356\3460\223\2739X\16c[\353\351\27\207y`#wUI\200\30\215\342\tD@\305;\244P\21\223\30cT4\306\244.\214\327\321[\27EA\255\276\254\320\324\241\224\23\272\243\215\215\236\\\2\364L\222F\306\317/5\352\363\353\376e[\327R\223\250\200\212#\21\2557\214\0074f2\277\214W\222\236\251\276\366/_Z5\324\335C\r\337\365O\345\0372\31\223\240\321a&\23\rW$\232\2039\230\304\2743\211\211\207h\"\227\354(\203\2539\31\245\213D\255\32\203KY*\226\273\32v<\200'\24\234\206\200:\24\34N\3sg\214I\314]\2652\7\23\315o\306\230?\250\321\34L\314d~3\321D\324\34*1\211\232\203\220;O9\230\250\321Lf2\275b4\232\203\332\6&\2321\252\352Z\231J\335\237\314\377R&!1\346\301@b\336\265\34\214\306\7\20D\353\245D\367n\27\221}\321\374fb&C\22\23\223\230\222F\352:t\350`i\335X\33\355:r2&)*Fk_\367\375V\251\253U\335\236
\204\340n\22c\3222\3716\310\4\33b\16\276C\245\266\206\325\232\323\220\201\323h\315\27"..., 1024, MSG_DONTWAIT, NULL, NULL) = 1024

recvfrom(3, "\366\237{\333\227\325w \236\260\374\245\267BMbE3ib\22\23'\252\27\232\377\3022w\342)\223\232\220\200IHh\344\240 &\221@bbN\22Z\22\343\224\304D4*1a\267U%o!=Uf\252\353 \211\211\271\0339\320\250\257=81\206ZR\330\332\323?\231;sg\3564\346\344!\211x\300Q\327\3228\4\23'0\7\242$\352\320\230\204K1*&\36\222\20\215!\2119H7\17\36@E\31\325z}\351[y{^\373\255\366\276\26\267'\361\200\216>\224}0T\4\225\341\350\260]\332\367\3575\30\353\377\265\313\377-\313\376\355\345\247\227\357\257U\333hO'\223\30\3030\7'\214\211fj\233\245\346\316\304Hi\242\335\230ht\230\223y0'\207\211\315$\0062Y\225Cw\312dL\275\324\333\327=\3762\300\337p\242\24\0255&\245&\236\22\343\204\212\342?\341\35\10\"\210\370_\340\tT\20b=\357JTP\243\250e\201\226\232\3\2713\357\214f\322\30\223x\210F\315\301D\243\306\207B\207`\265~\255\336j\332K\253\2270\324\2101\352P\324\341\350\30\22\215\342T\376\322\233\23\330\367\366\326\326e\355\237;~\4x\302\t\261\24<\200X[\355\330\326R\30\265\226SIkb\22\0235\236\242I\3240\24\234\222\230h\242\321L\246.\344\344)\7c\24\7\220\211\255\226^I\234\362\316\252Ds2\211\231 q\242\24jo\2jmU\265\213\265\2258D\35\353\17\21vZ\367\366wS\233\f'&-Ij]\315\364\346\233\211\265\224\267L\365\\m\307w\324\362\243\234\320\332\212\375R*\250\230\304\352(\273\32\247\264z)c4\250IL\324\304\34\314\301\210I4{\366\0305\335$\366\30\353\3635\264$N\361\16\5\2471\330v\335^
\327\227~"..., 1024, MSG_DONTWAIT, NULL, NULL) = 1024

recvfrom(3, "$&\326\225$N\303\356\30\177\371\371eI\312\234\254K\231\244\r\22sG\t&\365\372\332\274\276\226I}\256\372\374e\253\356\35~\204\342\4\376?\1N\345\224\1770\23\202&\325Kc>0\37hA\22\275T\215\241\10\325K\24\254\245\254\275/k\2414<\341T\350\366\372\332jy^\267q\373\327\245\372\227uo\2339XE#$\241\243(\216\326\220\266\225\377\204\247Z\312\273\201\232D\2439\231\23\321D\215\32s0\2472\306\244/$\306\324\313\216\230\304C&r\22\302n&\177\301\340\273j\245\16\25\37rg41'5&\32EsR\214QS\21S1\21\37Jl\202\350u\371\326\353\262\354\337\376\332\337\372V\375mm\273\211\211\212U\232\230\211=\267\317\335\34\0\25'h \356\36\352\355\346\333\322\25d\365\267R\360\16\307\326\232h\333\315\177!\25\207\211\23e\22MbN\346\240\325\t\33\211F\312\356\324\7]m\243\256\205\312\0A\333\325\312;5\2119\225\326J\222zz\322\304\34L\30\346\27\215\275\260\2721\7M<\345\0015\211\335\376\364\377\233\7\4\34C\300\251\226[\365\326\235\0204\377\3011\310\203\212y\7\266\321\252U\314D\243-U\35\343\7\370\33\326\245\251(\226\"\342Tk\325Z(U[\251 \226\271\323\374\301AB&\343?\325V\212\266*'\4\23\207\346`\314\37\20\304r\2\177CKQqB\314\301\2743\211\350J\222*\23aHUm\r\21\307\250\245\252m\265\265\262\252\215\336\252\360\35\36@\244\375x{\353K\tc}z2w\232\223\16\351d\"w\232\244\310\244\326\36s0\301\312\301\370\33\342Tk9\201\17\265\373\256\0Ox\300\t\247\3414\354\317\35E\21\251K\325
\272V\251Hm\342oU\242\240\250\303jZ\353\212\277\215\321\273\275k\355"..., 1024, 
MSG_DONTWAIT, NULL, NULL) = 1024

recvfrom(3, "?/}\271\254\243?\35\4'\234\272\16PP\341\266\224S_\324\30\202C\243\216\361w\321\211v\17\275u\305C\251\210\340C\201\312@s0\301XU\346@GzoU\246\312\230\223F\224\21\315Ic\222\206x\207\323\300\323\260\254\2\5\253\324\352K\331\236o\265\227Z\326@[-_\372\247N\177\331\315\7\304\360\2e&2Z\203\326\324\30\343\1\306@\223h\270\302K\253\16\236P\206\37\340poj\3\247\334\231\01741\202\247$\\I\345\27\0079\301(\210\377\0232pBM\342\37\222\f\25,'\274\312\306\245\252e\264>\354\325\250ku\n\" \230\377\312\200\211\30Q\222\30c,u\253\270\211\326\305\304\17\20\20\274\303m\3518\215\275\232x7<\264\336\233V\225\2126k\275\265j\232DL}\251D \261dl\273c\30k\335\251\336\264\375t\333\353\365m|]\366\332\227\365\371mY\177\334\332\247\3532\2321\211\207h\324D\2071\202\340\24Cm\344\216\32E\222\306\224f9\2524\7\215\211!\232\7\215Fp\352\0325>\324\353Zc\257\222\327\32{mV\27\260\323X\307\30\235\326\32\256\203\326X_\333\372\263\275l\205X\320@\253\177\251*\312Qm\373Q\5\375\373VN\345T\303i\10JGP\1kh\36LLb\16\6\214\207\352>\0J\341;\254V\356\377\336EK\304\211\356\324l\326\2728\206\310\320\267\v\306;P!*\3\n\261\266N\r%\202U}\364\221\242Z\353EUk\33\243\320m\23\177\253\255P\320Q\275\266B\35c\230\337L\204\244\206`\356\0245\272cD\342P\253s\333\3218\265\336\375\245{\302\23\26\36\30\214\226\332^\2671\306\217\365[\276\326s\377\334\267\177\337\352\266
\326\347\347\355\373\36249\355\226\214\246m\271\r\335^Z\25\265\324\266Q"..., 1024, 
MSG_DONTWAIT, NULL, NULL) = 1024

recvfrom(3, "\374\5\17\250\330Z]\227\25a\214~]\313\321D\35V[\212\376\322q\214\201\356N\255us\347\224\270\232Z\267u\214\376\246}+\333\376\23\17\250Pki-}\327j\333\262\337\336\272\373\323\304@\304\335\332:\265\364\275\336Z\365ei\325\255V\373^\326\317\275\226^\275\277}\177\351\333\365\323\322\237k\214z\276\254\317\327\255\217a[\273\20\243\345\35\336\201\345\330\255\352\303\34\2149\325\332Z\265\326\250\32\5\255\265\261wc>\320\230\0301w\306\207\265mU\227R\34Nc\260\226\207z+?\200\372\321\252<\200(\210\226\250M\301\3464@\35JG\305\321\235\20Qa\7\24\35\303w\340;p\210\366\267\325\tA\234\330i\356\326\217\355m\33\243\350\317\r\255\255/\375\326\373\250u\357\365\372j\16b\22\v5\17\224&1T\351\30\250\210\34\334G-mT\377R\212Z&!\361\326\332kK\flU$q\253\4\211w\340\204w8\321\1\265v\337\201\310x\353\236\300\2076LB\16\226IL\304\330[\31G\31b\356\264Lb>\262\25\2512\377`\221\3\303\30\207\304|DA\253Q\205\210\330{U9\355\0360\336%Q0\3410\374\23N8\225\250\10\342a\f\301Z\313\3U\205x\300^\345M\367\265\306\30I\217I\306p\370\227\366~\361\257a\2519\30\2*I\f\230p%\211F\315\244&\304)1'\223hPs\"\277\325F\22\267u\3444l\25\207\275\211V\323Lu-Z\257\361ioT\231?\231I\210\325\250\242<E\203\"N\305\265\214i\243#Hk>\240\202\260\367/\27k[Z\333kh\253V\232\3\321x\312\344\1\3F<\24\345a\250\335amUk\201\"\325\360\237\300;d\370\321\20\357\300\273\22\20\321$$\301
\210\207$\242\230\304Dk\224\225\301\376\272\361"..., 1024, MSG_DONTWAIT, NULL, NULL) = 
1024

recvfrom(3, "\231\23I\10eb\22\32\240`m\205\277 :@E\21OU\315\361\251\214X\345\324\n\245\253\221\352\342\350\335\23\266K\311\300\261\357\250\265u\267\275\264\256\205\271\263HtX5X\200\322\34\24\223\f\201\241\336\212\272\322\333\245\325`T\222\276l\335w\200\370\200wm+\4\247\22\313\3\3506\372VUC\252\255E\265\346\204\342;TPD\252\227\340f\336i&\203}\355\267jo\343r\31\337^\273y\347 \211%e\264L\r\225m\363P\2Z\205}\371v[?ui\324V\nb\356\214I\234\206\242I\214\30\211'T\224\2418!>\0\345\237(\377\1\337\1\26T[.\255~\\\326\275\2665\320\277\277\365\315\17:\376R\227r\252^\376\3\310\240\336\352\355\366\375\355\265_\226V\353\337\213\230\304(\211R&&%\30\35\345\270j~\253\352\356/\235\3536ZiHu\6\346db\342!\7\243MC\341!\211qJ\242e\342C\214\275\24EA\35\265U]j\224\265\324^[a=\367\347\256&\361\220x\312dt\2408\341\377\260:\225\26\345TPj-U{9Q\255\300\252V\n\346\3\23\315\7x\30\276\212U\255\264v\350\257\275j\255\347\347\336\227\352U[U\231_\204DH\210\2\232\304*s0hT\320\204\210\207\250\375\2478\1\205:\272X[\t:,j+\247\374\302\337\344d\273\364\347\376\274m\215\252\r\315\301\374A\324M3\31sR\350`\2u\253$\343e\217\241^[9\341\35\212\2771<a\376\7:9h4\17\306CLb44\252\227\23>\364^\36@E\300\3j>2'1\223\32\0376\207H\255\345p\f<\201(Ue\16\16\315\377\"?\265\372\227
\336\301vq\355\257\267\257\313\253\364\313\362\372\257\177_o\237\227\361S\315dL\242\201
\2501\306\30"..., 1024, MSG_DONTWAIT, NULL, NULL) = 1024

recvfrom(3, "9h6\247\265uumk\333?\367\376\332FG<4|\350\242\16\265\17:\216>\372\247\356E\35}\321MF\337\235\30\210v]\333\20\17\f\177\351\205\212 J\355\2128\372\360\216&\202\256\177\377\334\252oK\177^\314\377\3\203\304\341o\340o8\272\210V\363\204J]J\353\322\274\3\17\325\312\t\355?Q\313\335U\34\200}+\31\273\3\177CQ\4\245]j\307\241}/AE\24\334{]\332R}\277\374u\273^\n\353\355m)<\0\322\350b!\212\212\356B\251\231\214\231L\314\203\251\245\256[\273\265W\315d\214\321xHB+ML$\232\23e4bN8^\273J\377\336\314\7&1\377\340\224\7\223H\314Tk\245\236Gb\2\236\30\376\7\304\23\202C\240\266\322\321\330\241\377\370\327\227K__^.\317\317oOOO\346\316L&\346\27\r\202\335\207QNhm\5>\2402\274\303\201\370\21\10\nh\22\267\212F\355-&\241\3\16\255\16NI4\232h\314\t\345\2\35\17\271\363\220\250\271\323\242\6\306)\7\363\237\314\311;\246\256\326V\342\4j\255\227r\32\16\266\306B\353@\252oc\273\260\334\252./\227\235\215a{U\223\370\v\202O\177\"$a+2\276\256\315\267\315d\354\327\327\313\353u\273\\\277^Vc7\211I\3122\23+\231\240\2130\276\f\223X\244\256\225\226\251\3555\256\324\353_{O\322\256?\203\346O\346\37\214\346d\214;fr\2308*j+4\357z\350\205\346\316\262z%\32\233\357:\326hM\353VC\240\277\336\326\266\274<\231\223&Z\225\223\261\267\327\345\366\366\255\261nO\306$FKc4\7\215\253\30\35\232\7\0076#\r62\211\2261~\4((*U\245\330Js\247I4\7\313
\30\223\210U\346\37\f\235\34h0*\16\315"..., 1024, MSG_DONTWAIT, NULL, NULL) = 993

recvfrom(3, 0x7ffebf99ffc0, 1024, 64, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)

shutdown(3, 0 /* receive */)            = 0

close(3)                                = 0

Unable to stream SRR12596531 but can read from local disk

I would like to read all SRA records using AWS instances that do not have any local storage. To this end, I would like to stream the contents of each SRA record directly from its home in AWS S3 to my application using the NCBI SRA C++ API (e.g., ngs::ReadIterator).

In general, this works well. However, when testing my algorithm against some of the largest SRA records, the following error is reproducibly encountered when streaming SRR12596531 after reading about 5.7% of the spots:

VCursorCellDataDirect failed: '(INSDC:dna:text)READ' [188973057] rc = RC(rcXF,rcFunction,rcExecuting,rcParam,rcBadVersion)

This same SRA record (SRR12596531) can be read from local storage (after downloading with prefetch) without any error.

While the above mentioned error is being generated by unzip_func() in ncbi_vdb/libs/vxf/unzip.c I suspect that the error is caused by passing invalid VBlobHeader information to the unzip function. Any suggestions for tracking down this issue would be appreciated! The use of functions that are assigned at run-time makes the code very difficult to read. Is there a way to output a human-readable function call stack when an error is encountered?

Install can't find libxml2

Hi,

It seems that the configure script accepts --with-xml2-prefix=dir but then fails to put dir/lib in the library search path. This leads to the following link error:

ln -s libvdb-sqlite.a libvdb-sqlite-static.a
gcc -shared -o /home/br51/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/lib/libvdb-sqlite.so.2.10.7 -Wl,-soname,libvdb-sqlite.so.2 -DNDEBUG  -m64  sqlite3.pic.o sqlite3vdb.pic.o -L/home/br51/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/lib -L/home/br51/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/ilib -Wl,-Bstatic -Wl,-whole-archive -lncbi-vdb -Wl,-no-whole-archive -Wl,-Bdynamic -ldl -lpthread -lxml2 -lm
/usr/bin/ld: cannot find -lxml2
collect2: error: ld returned 1 exit status
make[4]: *** [/home/br51/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel/lib/libvdb-sqlite.so] Error 1
make[3]: *** [std] Error 2
make[2]: *** [vdb-sqlite] Error 2
make[1]: *** [libs] Error 2
make: *** [install] Error 2

In build/ld.linux.dlib.sh we have:

# add in xml                                                                                             
if [ $HAVE_XML -ne 0 ]
then
CMD="$CMD -lxml2
fi

It seems that you would need to change this to -L <xml2-prefix>/lib -lxml2 in order for this to work.

-BenRI

make -j4 fails

make -j1 does however succeed.

mv: rename dna-reverse-cmpl.pic.o to align-reader-dna-reverse-cmpl.pic.o: No such file or directory
mv: rename reference-cmn.pic.o to align-reader-reference-cmn.pic.o: No such file or directory
…
mv: rename align-ref-name.pic.o to axf-align-ref-name.pic.o: No such file or directory
mv: rename align-ref-seq-id.pic.o to axf-align-ref-seq-id.pic.o: No such file or directory

For a complete log, see https://gist.githubusercontent.com/sjackman/ba0838ed694a09007da2a09d6d355c41/raw/afec628ebc5df61f5c703da2545f3090699dad09/05.make
and https://gist.github.com/sjackman/ba0838ed694a09007da2a09d6d355c41

Test for root in make install

Hello,

We are getting errors on bioconda (bioconda/bioconda-recipes#8525) because of the test for ROOT at the begining of build/Makefile.install.

The build is detected as being run by the root and it tries to link the libraries to a folder that does not existe. The faulting part is this one :

# set up installation locations

#fake root for debugging
#uncomment this line and change the following test for root ( see under install: ) to succeed:
#ROOT = ~/root
ifeq (linux, $(OS))
    ifeq (0, $(shell id -u))
        LINUX_ROOT = true
    endif
endif

ifeq (true, $(LINUX_ROOT))
    KONFIG_DIR = $(ROOT)/etc/ncbi
    INCLUDE_SYMLINK = $(ROOT)/usr/include/ncbi-vdb
    PROFILE_FILE = $(ROOT)/etc/profile.d/ncbi-vdb
else
    KONFIG_DIR = $(LIB_TARGET)/ncbi
endif

It says it's only for debugging and we should change the test. However this is not convinient because it means we have to patch the file in the bioconda build.

Do you think we could get a fix on this ? I could submit a PR but I'm not sure what should be done.

(there is the exact same problem in ngs-tools)

Construction method of arm platform

How to build sratools under arm architecture？

Extreme memory consumption when reading certain SRA records?

I am reading sequence data from SRA records by first downloading the SRA record with the prefetch command and then iterating through the file using the C++ interface (version 2.10.8), i.e.:

ngs::ReadCollection run("DRR001375");
const size_t num_read = run.getReadCount(ngs::Read::all);
ngs::ReadIterator run_iter = ngs::ReadIterator( run.getReadRange ( 1, num_read, ngs::Read::all ) );

size_t read_count = 0;
while( run_iter.nextRead() ){

	++read_count;
	while( run_iter.nextFragment() ){
		
		const string seq = run_iter.getFragmentBases().toString();

		// Process the read sequence ...
		process_read(seq);
	}
}

In general, this approach seems to work well. However, I have noticed that for some SRA records (like ERR191522), there is (a) significant memory consumption and (b) a dramatic slow-down when iterating through the file. The following plot shows the speed (in reads per second) and memory consumption (from /proc/meminfo, reported as a fraction of total system memory):

Other SRA records seem to be fine. For DRR001375, the following graph shows fairly constant speed and memory usage:

Is there a way to read SRA records, like ERR191522, without the large memory consumption? If not, is there a way to identify SRA records (in advance) that will exhibit this behavior (as the available RAM on on cluster instances can easily be exhausted while processing a single SRA record).

Reduce maximum memory usage while reading SRA data?

Using the C VDB API (and following the fasterq-dump utility strategy for accessing SRA records) for reading SRA data can consume a significant amount of RAM while reading an SRA record. This can be an issue when using attempting to minimize the amount of Cloud computing resources (i.e. instance RAM) when processing a large number of SRA records.

The maximum amount of RAM used while reading (as measured with /usr/bin/time -v) depends on the record:

While periodically calling VCursorRelease() and VCursorOpen() to force the VDB interface to deallocate RAM offers a minor reduction in the maximum amount of RAM used (about 25%), this strategy significantly slows down the rate at which an SRA record is read.

Is it possible/feasible to limit memory consumption using the VDB C API to sub-gigabyte levels, independent of the number of reads? The goal is to read through an SRA record once, as quickly as possible and using as little RAM as possible.

Can't find hdf5 after apt-get installation

Hello,

I'm trying to setup vdb on Ubuntu 14.04 and ran into the following:

make install
Checking make status of object libraries...
NOTE - library libkdf5 cannot be built: It requires 'libhdf5' and its development headers.
Installing libraries to /usr/local/ncbi/ncbi-vdb/lib64
Installing includes to /usr/local/ncbi/ncbi-vdb/include
Installing configuration files to /etc/ncbi
Updating /usr/include/ncbi-vdb
Updating /etc/profile.d/ncbi-vdb.[c]sh
Use $NCBI_VDB_LIBDIR in your link commands, e.g.:
      ld -L$NCBI_VDB_LIBDIR -lncbi-vdb ...
Please add /usr/local/ncbi/ncbi-vdb/lib64 to your LD_LIBRARY_PATH, e.g.:
      export LD_LIBRARY_PATH=/usr/local/ncbi/ncbi-vdb/lib64:$LD_LIBRARY_PATH
Use /usr/local/ncbi/ncbi-vdb/lib64 in your link commands, e.g.:
      export NCBI_VDB_LIBDIR=/usr/local/ncbi/ncbi-vdb/lib64
      ld -L$NCBI_VDB_LIBDIR -lncbi-vdb ...

Going back to the configure step yields the following

./configure
Configuring NCBI-VDB package
checking system type... Linux
checking OS distributor... Ubuntu
checking machine architecture... x86_64
checking for supported architecture... x86_64 (64 bits) is supported
checking for supported OS... Linux (linux) is supported
checking for supported tool chain... gcc tool chain is supported
checking for g++... yes
checking whether gcc accepts -Wno-array-bounds... yes
checking whether g++ accepts -static-libstdc++... no
checking for fuse library... yes
checking for hdf5 library... no
checking for magic library... yes
checking for xml2 library... yes
checking for ngs-sdk package...
	includes... ../ngs/ngs-sdk
	includes... /usr/local/ngs/ngs-sdk
	libraries... /usr/local/ngs/ngs-sdk/lib64
includes: /Apps/ngs/ngs-sdk
libraries: /usr/local/ngs/ngs-sdk/lib64
checking for ngs-java package...
	jar... /usr/local/ngs/ngs-java/jar/ngs-java.jar
jar: /usr/local/ngs/ngs-java/jar/ngs-java.jar.2.9.2

configure: creating 'build/ld.linux.exe_cmd.sh'
configure: creating 'build/Makefile.config.linux.x86_64'
configure: creating 'Makefile.config'
configure: creating 'reconfigure'
build type: release
build prefix: /root/ncbi-outdir/ncbi-vdb
build output path: /root/ncbi-outdir/ncbi-vdb/linux/gcc/x86_64/rel
includedir: /usr/local/ncbi/ncbi-vdb/include
bindir: /usr/local/ncbi/ncbi-vdb/bin
libdir: /usr/local/ncbi/ncbi-vdb/lib
CC = gcc -c
CPP = g++
configured with: ""

However I have hdf5-dev installed:

apt-get install libhdf5-dev libhdf5-serial-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
libhdf5-dev is already the newest version.
libhdf5-serial-dev is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 162 not upgraded.

To verify:

ld -lhdf5 --verbose
...
==================================================
attempt to open /usr/x86_64-linux-gnu/lib64/libhdf5.so failed
attempt to open /usr/x86_64-linux-gnu/lib64/libhdf5.a failed
attempt to open //usr/local/lib/x86_64-linux-gnu/libhdf5.so failed
attempt to open //usr/local/lib/x86_64-linux-gnu/libhdf5.a failed
attempt to open //usr/local/lib64/libhdf5.so failed
attempt to open //usr/local/lib64/libhdf5.a failed
attempt to open //lib/x86_64-linux-gnu/libhdf5.so failed
attempt to open //lib/x86_64-linux-gnu/libhdf5.a failed
attempt to open //lib64/libhdf5.so failed
attempt to open //lib64/libhdf5.a failed
attempt to open //usr/lib/x86_64-linux-gnu/libhdf5.so succeeded
-lhdf5 (//usr/lib/x86_64-linux-gnu/libhdf5.so)
libpthread.so.0 needed by //usr/lib/x86_64-linux-gnu/libhdf5.so
found libpthread.so.0 at //lib/x86_64-linux-gnu/libpthread.so.0
libz.so.1 needed by //usr/lib/x86_64-linux-gnu/libhdf5.so
found libz.so.1 at //lib/x86_64-linux-gnu/libz.so.1
libdl.so.2 needed by //usr/lib/x86_64-linux-gnu/libhdf5.so
found libdl.so.2 at //lib/x86_64-linux-gnu/libdl.so.2
libm.so.6 needed by //usr/lib/x86_64-linux-gnu/libhdf5.so
found libm.so.6 at //lib/x86_64-linux-gnu/libm.so.6
libc.so.6 needed by //usr/lib/x86_64-linux-gnu/libhdf5.so
found libc.so.6 at //lib/x86_64-linux-gnu/libc.so.6
ld-linux-x86-64.so.2 needed by //lib/x86_64-linux-gnu/libpthread.so.0
found ld-linux-x86-64.so.2 at //lib/x86_64-linux-gnu/ld-linux-x86-64.so.2

I retried after exporting /usr/lib/x86_64-linux-gnu to LD_LIBRARY_PATH, as I read it is a non-standard library location, yet the messages are identical.
Sorry for the edits, I didn't preview before posting.

`with-vdb explicitly specified, but not supported on this platform` error

I am on Ubuntu 16.04 32 bit platform. I am trying to build gbench-2.12.0. I ran build-linux-distro.sh
I applied this patch to bypass msse2 failure. It compiled and installed ncbi-vdb locally, and the script went on but it failed with with-vdb explicitly specified, but not supported on this platform error in the final /configure process.

I checked out the configure parameters.

./configure --with-dll --with-mt --with-gbench --with-vdb=/home/***/GENOME/NCBI-TOOLS/gbench-src-2.12.0/ThirdParty/i386Release --without-debug --with-boost --with-libxml --with-libxslt --with-sqlite3 --with-lzo --with-ftgl --with-gnutls --with-wxwidgets=/home/***/GENOME/NCBI-TOOLS/gbench-src-2.12.0/ThirdParty/i386Release --with-wxwidgets-ucs --with-projects=scripts/projects/ncbi_gbench.lst --with-runpath=/opt/ncbi/gbench-2.12.0/lib --without-sybase --with-flat-makefile --without-caution

Why is it failing?
I hope you guide me.
Regards
Kenn

Error encountered while compiling for Arm platform

I got an error when compiling on arm platform

Consolidate compiler generated dependencies of target ncbi-bam
[  0%] Building C object libs/align/CMakeFiles/ncbi-bam.dir/bam.c.o
In file included from /opt/installer/ncbi-vdb-3.0.0/interfaces/kfs/file-v2.h:35:0,
                 from /opt/installer/ncbi-vdb-3.0.0/interfaces/kfs/file.h:39,
                 from /opt/installer/ncbi-vdb-3.0.0/libs/align/bam.c:32:
/opt/installer/ncbi-vdb-3.0.0/interfaces/kfc/refcount.h:39:10: fatal error: atomic32.h: No such file or directory
 #include <atomic32.h>
          ^~~~~~~~~~~~
compilation terminated.

I guess it should be due to the wrong directory name. On arm, env.cmake includes interfaces/cc/gcc/aarch64 but what actually exists is interfaces/cc/gcc/arm64.

ncbi-vdb/build/env.cmake

Lines 68 to 71 in 34c6113

    
           elseif ( ${CMAKE_HOST_SYSTEM_PROCESSOR} STREQUAL "arm64") 
        
               set(ARCH "aarch64") 
        
           elseif ( ${CMAKE_HOST_SYSTEM_PROCESSOR} STREQUAL "aarch64") 
        
               set(ARCH "aarch64")

ncbi-vdb/build/env.cmake

Lines 158 to 160 in 34c6113

    
           if ( "GNU" STREQUAL "${CMAKE_C_COMPILER_ID}") 
        
               include_directories(interfaces/cc/gcc) 
        
               include_directories(interfaces/cc/gcc/${ARCH})

I renamed interfaces/cc/gcc/arm64 to interfaces/cc/gcc/aarch64 and everything looks fine.

Build system doesn't find cmake3 on CentOS 7

CentOS 7 has both cmake version 2.8.x and cmake3 version 3.17.x available in the system packages. The build system fails because it only finds the older cmake:

CMake Error at CMakeLists.txt:52 (cmake_minimum_required):
  CMake 3.16 or higher is required.  You are running version 2.8.12.2

Other than hacky solutions like removing the system's cmake 2.8.x and symlinking cmake to the cmake3 binary, how can I tell the build system where to find cmake3?

Please remove explicit -m32/-m64 switch

Hi,

I received a bug report for the Debian package and this patch solves the issue. Please note that there are strong opinions that -m32 and -m64 switches are doing more harm than good in general.

Kind regards

     Andreas.

undefined reference to `__stack_chk_guard' when configuring IgBlast

I'm trying to compile IgBlast 1.17.1 on a new HPC system. During configure, it builds ncbi-vdb but is unable to link it

configure: trying to build the NCBI SRA/VDB Toolkit from GitHub.
checking for ncbi-vdb in /work/01114/vdj/ls6/igblast/1.17.1/tmp/ncbi-igblast-1.17.1-src/c++/local/ncbi-vdb-2.10.8... no
configure: error: --with-vdb explicitly specified, but no usable version found.

when looking in config.log, I see the error

configure:26826: checking for ncbi-vdb in /work/01114/vdj/ls6/igblast/1.17.1/tmp/ncbi-igblast-1.17.1-src/c++/local/ncbi-vdb-2.10.8
configure:26843: /opt/apps/gcc/11.2.0/bin/g++  -std=gnu++14 -msse4.2 -o conftest  -Wall -Wno-format-y2k  -pthread -fopenmp -O2 -fPIC  -DNDEBUG -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_L
ARGEFILE64_SOURCE   -D_MT -D_REENTRANT -D_THREAD_SAFE -I/work/01114/vdj/ls6/igblast/1.17.1/tmp/ncbi-igblast-1.17.1-src/c++/local/ncbi-vdb-2.10.8/include -I/work/01114/vdj/ls6/igblast/1.17.1/
tmp/ncbi-igblast-1.17.1-src/c++/local/ncbi-vdb-2.10.8/include/os/linux -I/work/01114/vdj/ls6/igblast/1.17.1/tmp/ncbi-igblast-1.17.1-src/c++/local/ncbi-vdb-2.10.8/include/os/unix -I/work/0111
4/vdj/ls6/igblast/1.17.1/tmp/ncbi-igblast-1.17.1-src/c++/local/ncbi-vdb-2.10.8/include/cc/gcc/x86_64 -I/work/01114/vdj/ls6/igblast/1.17.1/tmp/ncbi-igblast-1.17.1-src/c++/local/ncbi-vdb-2.10.
8/include/cc/gcc  -Wl,-rpath,/opt/apps/gcc/11.2.0/lib/gcc/x86_64-pc-linux-gnu/11.2.0/../../../../lib64 -Wl,--enable-new-dtags -Wl,-export-dynamic  -pthread -fopenmp   -O2 conftest.cpp   -ldl
 -lm  -lpthread -L/work/01114/vdj/ls6/igblast/1.17.1/tmp/ncbi-igblast-1.17.1-src/c++/local/ncbi-vdb-2.10.8/lib64 -Wl,-rpath,/work/01114/vdj/ls6/igblast/1.17.1/tmp/ncbi-igblast-1.17.1-src/c++
/local/ncbi-vdb-2.10.8/lib64 -lncbi-vdb   -lz    -lz  -ldl >&5
/opt/apps/gcc/11.2.0/bin/ld: /work/01114/vdj/ls6/igblast/1.17.1/tmp/ncbi-igblast-1.17.1-src/c++/local/ncbi-vdb-2.10.8/lib64/libncbi-vdb.so: undefined reference to `__stack_chk_guard'
collect2: error: ld returned 1 exit status

Here is system and gcc information:

$ lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:	Rocky
Description:	Rocky Linux release 8.4 (Green Obsidian)
Release:	8.4
Codename:	GreenObsidian

$ gcc --version
gcc (GCC) 11.2.0

I tried configuring IgBlast with the Intel compilers but I still get the same error; I think that ncbi-vdb is still being compiled with gcc.

As a workaround, I modified build/Makefile.gcc and deleted the -fstack-protector option, then recompiled. That let the IgBlast configure finish, but I'm not sure if that's going to cause problems during the build or at runtime.

Installs static libs instead of shared libs, for example lib/libncbi-vdb.a.2.11.0

Version 2.11.0 installs static libs instead of shared libs:

lib/libkdf5-static.a
lib/libkdf5.a
lib/libkdf5.a.2 
lib/libkdf5.a.2.11.0
lib/libkdf5.so
lib/libkdf5.so.2
lib/libkdf5.so.2.11.0
lib/libncbi-ngs-c++-static.a
lib/libncbi-ngs-c++.a
lib/libncbi-ngs-c++.a.2
lib/libncbi-ngs-c++.a.2.11.0
lib/libncbi-ngs-static.a
lib/libncbi-ngs.a
lib/libncbi-ngs.a.2
lib/libncbi-ngs.a.2.11.0
lib/libncbi-vdb-static.a
lib/libncbi-vdb.a
lib/libncbi-vdb.a.2
lib/libncbi-vdb.a.2.11.0
lib/libncbi-vdb.so
lib/libncbi-vdb.so.2
lib/libncbi-vdb.so.2.11.0
lib/libncbi-wvdb-static.a
lib/libncbi-wvdb.a
lib/libncbi-wvdb.a.2
lib/libncbi-wvdb.a.2.11.0
lib/libncbi-wvdb.so
lib/libncbi-wvdb.so.2
lib/libncbi-wvdb.so.2.11.0

This issue is similar to issue ncbi/ngs#33 in the ngs project.

OS: FreeBSD 12.2

build issues with gcc 10.3.0 on Linux/x86-64

Hi, I'm trying to build ncbi-vcb 2.11.2 on a CentOS 7 box with pkgsrc-2021Q2 for various updated dependencies. I was able to build 2.10.8 in the same environment, but the current release/tag just doesn't want to play because of various issues.

Using std::malloc and friends without cstdlib header: I need to add `include ' to vdb3/platform/memory/PrimordialMemoryMgr.cpp (and added std:: prefixes, of which I am not sure if you need them there).
Unpleasant warnings … even if the build succeeded, I would be worried about them (more below)
Undefined references at the end of the build:

[ 98%] Linking CXX executable /scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/.build/ncbi-vdb/linux/gcc/x86_64/rel/test-bin/test-cmn-log
/sw/compiler/gcc-10.3.0/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/test-cmn-nconv.dir/nconv-tst.cpp.o: in function `vdb3::NumberConversionTest_DecFltPoint64_from_int_to_Text_Test::TestBody()':
nconv-tst.cpp:(.text+0x123ed): undefined reference to `testing::internal::CmpHelperEQ(char const*, char const*, long long, long long)'
/sw/compiler/gcc-10.3.0/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: nconv-tst.cpp:(.text+0x12446): undefined reference to `testing::internal::CmpHelperEQ(char const*, char const*, long long, long long)'
/sw/compiler/gcc-10.3.0/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/test-cmn-nconv.dir/nconv-tst.cpp.o: in function `vdb3::NumberConversionTest_DecFltPoint64_from_int_to_String_Test::TestBody()':
nconv-tst.cpp:(.text+0x12acd): undefined reference to `testing::internal::CmpHelperEQ(char const*, char const*, long long, long long)'
/sw/compiler/gcc-10.3.0/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: nconv-tst.cpp:(.text+0x12b25): undefined reference to `testing::internal::CmpHelperEQ(char const*, char const*, long long, long long)'
/sw/compiler/gcc-10.3.0/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: nconv-tst.cpp:(.text+0x12bf9): undefined reference to `testing::internal::CmpHelperEQ(char const*, char const*, long long, long long)'
/sw/compiler/gcc-10.3.0/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/test-cmn-nconv.dir/nconv-tst.cpp.o:nconv-tst.cpp:(.text+0x12c51): more undefined references to `testing::internal::CmpHelperEQ(char const*, char const*, long long, long long)' follow
/sw/compiler/gcc-10.3.0/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: CMakeFiles/test-cmn-nconv.dir/nconv-tst.cpp.o: in function `testing::AssertionResult testing::internal::CmpHelperEQFailure<vdb3::CTextLiteral, vdb3::CText>(char const*, char const*, vdb3::CTextLiteral const&, vdb3::CText const&)':
nconv-tst.cpp:(.text._ZN7testing8internal18CmpHelperEQFailureIN4vdb312CTextLiteralENS2_5CTextEEENS_15AssertionResultEPKcS7_RKT_RKT0_[_ZN7testing8internal18CmpHelperEQFailureIN4vdb312CTextLiteralENS2_5CTextEEENS_15AssertionResultEPKcS7_RKT_RKT0_]+0x1bc): undefined reference to `testing::internal2::PrintBytesInObjectTo(unsigned char const*, unsigned long, std::ostream*)'
/sw/compiler/gcc-10.3.0/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: nconv-tst.cpp:(.text._ZN7testing8internal18CmpHelperEQFailureIN4vdb312CTextLiteralENS2_5CTextEEENS_15AssertionResultEPKcS7_RKT_RKT0_[_ZN7testing8internal18CmpHelperEQFailureIN4vdb312CTextLiteralENS2_5CTextEEENS_15AssertionResultEPKcS7_RKT_RKT0_]+0x446): undefined reference to `testing::internal2::PrintBytesInObjectTo(unsigned char const*, unsigned long, std::ostream*)'
collect2: error: ld returned 1 exit status
make[4]: *** [platform/cmn/test/CMakeFiles/test-cmn-nconv.dir/build.make:100: /scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/.build/ncbi-vdb/linux/gcc/x86_64/rel/test-bin/test-cmn-nconv] Error 1
make[4]: Leaving directory '/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/.build/ncbi-vdb/linux/gcc/x86_64/rel/vdb3'
make[3]: *** [CMakeFiles/Makefile2:1325: platform/cmn/test/CMakeFiles/test-cmn-nconv.dir/all] Error 2
make[4]: Leaving directory '/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/.build/ncbi-vdb/linux/gcc/x86_64/rel/vdb3'
[ 98%] Built target test-cmn-log
make[3]: Leaving directory '/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/.build/ncbi-vdb/linux/gcc/x86_64/rel/vdb3'
make[2]: *** [Makefile:146: all] Error 2
make[2]: Leaving directory '/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/.build/ncbi-vdb/linux/gcc/x86_64/rel/vdb3'
make[1]: *** [Makefile:33: cmake] Error 2
make[1]: Leaving directory '/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/vdb3'
make: *** [Makefile:51: vdb3] Error 2

Does that ring a bell? Testing::internal not being in the mix? I'm pausing right now, but maybe there is an easy fix awaiting me when I come back …

The warnings:

/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/ext/magic/apprentice.c:2276:3: warning: ignoring return value of ‘asprintf’ declared with attribute ‘warn_unused_result’ [-Wunused-result]
 2276 |   asprintf(&buf, "%.*s.mime%s", (int)(q - fn), fn, ext);
      |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/ext/magic/apprentice.c:2283:2: warning: ignoring return value of ‘asprintf’ declared with attribute ‘warn_unused_result’ [-Wunused-result]
 2283 |  asprintf(&buf, "%.*s%s", (int)(q - fn), fn, ext);
      |  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/ext/magic/compress.c: In function ‘uncompressbuf’:
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/ext/magic/compress.c:394:14: warning: ignoring return value of ‘dup’ declared with attribute ‘warn_unused_result’ [-Wunused-result]
  394 |       (void) dup(fd);
      |              ^~~~~~~
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/ext/magic/compress.c:397:14: warning: ignoring return value of ‘dup’ declared with attribute ‘warn_unused_result’ [-Wunused-result]
  397 |       (void) dup(fdin[0]);
      |              ^~~~~~~~~~~~
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/ext/magic/compress.c:403:10: warning: ignoring return value of ‘dup’ declared with attribute ‘warn_unused_result’ [-Wunused-result]
  403 |   (void) dup(fdout[1]);
      |          ^~~~~~~~~~~~~
In function ‘close_and_restore’,
    inlined from ‘file_or_fd’ at /scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/ext/magic/magic.c:370:2,
    inlined from ‘magic_descriptor’ at /scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/ext/magic/magic.c:269:9:
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/ext/magic/magic.c:249:10: warning: argument 1 null where non-null expected [-Wnonnull]
  249 |   (void) utimes(name, utsbuf); /* don't care if loses */
      |          ^~~~~~~~~~~~~~~~~~~~
In file included from /scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/ext/magic/magic.c:47:
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/ext/magic/magic.c: In function ‘magic_descriptor’:
/usr/include/sys/time.h:139:12: note: in a call to function ‘utimes’ declared here
  139 | extern int utimes (const char *__file, const struct timeval __tvp[2])
      |            ^~~~~~
In function ‘file_or_fd’,
    inlined from ‘magic_descriptor’ at /scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/ext/magic/magic.c:269:9:
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/ext/magic/magic.c:359:4: warning: ‘%s’ directive argument is null [-Wformat-overflow=]
  359 |    file_error(ms, errno, "cannot read `%s'", inname);
      |    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Well, that's not your code, but when you vendor it, it becomes your problem … Then, we got:

/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyIns.c: In function ‘j__udyInsWalk’:
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyIns.c:1504:51: warning: iteration 8 invokes undefined behavior [-Waggressive-loop-optimizations]
 1504 |                 JU_BITMAPSETL(Pjlb, Pjp->jp_1Index[offset]);
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/Judy1.h:400:58: note: in definition of macro ‘J1_JLB_BITMAP’
  400 | #define J1_JLB_BITMAP(Pjlb,Subexp)  ((Pjlb)->j1lb_Bitmap[Subexp])
      |                                                          ^~~~~~
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyPrivate.h:822:26: note: in expansion of macro ‘JU_SUBEXPL’
  822 |     (JU_JLB_BITMAP(PJLB, JU_SUBEXPL(INDEX)) |= JU_BITPOSMASKL(INDEX))
      |                          ^~~~~~~~~~
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyIns.c:1504:17: note: in expansion of macro ‘JU_BITMAPSETL’
 1504 |                 JU_BITMAPSETL(Pjlb, Pjp->jp_1Index[offset]);
      |                 ^~~~~~~~~~~~~
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyIns.c:1503:13: note: within this loop
 1503 |             for (offset = 0; offset < 15; ++offset)
      |             ^~~
In file included from /scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/Judy1.h:70,
                 from /scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyIns.c:29:
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyPrivate.h:1096:31: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
 1096 |         (PBYTE)[i_dx + 0 + 5] = (PBYTE)[i_dx + 0];              \
      |         ~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyIns.c:1235:13: note: in expansion of macro ‘JU_INSERTINPLACE5’
 1235 |             InsertInPlace(Pjll, exppop1, offset, Index);                \
      |             ^~~~~~~~~~~~~
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyIns.c:1592:13: note: in expansion of macro ‘JU_IMMSETINPLACE’
 1592 |             JU_IMMSETINPLACE(5, uint8_t *, cJ1_JPIMMED_5_02, j__udySearchLeaf5,
      |             ^~~~~~~~~~~~~~~~
In file included from /scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/Judy1.h:71,
                 from /scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyIns.c:29:
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyPrivateBranch.h:81:21: note: at offset 0 to object ‘j_pi_1Index’ with size 8 declared here
   81 |             uint8_t j_pi_1Index[sizeof(Word_t)];        // see above.
      |                     ^~~~~~~~~~~
In file included from /scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/Judy1.h:70,
                 from /scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyIns.c:29:
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyPrivate.h:1097:31: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
 1097 |         (PBYTE)[i_dx + 1 + 5] = (PBYTE)[i_dx + 1];              \
      |         ~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyIns.c:1235:13: note: in expansion of macro ‘JU_INSERTINPLACE5’
 1235 |             InsertInPlace(Pjll, exppop1, offset, Index);                \
      |             ^~~~~~~~~~~~~
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyIns.c:1592:13: note: in expansion of macro ‘JU_IMMSETINPLACE’
 1592 |             JU_IMMSETINPLACE(5, uint8_t *, cJ1_JPIMMED_5_02, j__udySearchLeaf5,
      |             ^~~~~~~~~~~~~~~~
In file included from /scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/Judy1.h:71,
                 from /scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyIns.c:29:
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyPrivateBranch.h:81:21: note: at offset 0 to object ‘j_pi_1Index’ with size 8 declared here
   81 |             uint8_t j_pi_1Index[sizeof(Word_t)];        // see above.
      |                     ^~~~~~~~~~~
In file included from /scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/Judy1.h:70,
                 from /scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyIns.c:29:
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyPrivate.h:1098:31: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
 1098 |         (PBYTE)[i_dx + 2 + 5] = (PBYTE)[i_dx + 2];              \
      |         ~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyIns.c:1235:13: note: in expansion of macro ‘JU_INSERTINPLACE5’
 1235 |             InsertInPlace(Pjll, exppop1, offset, Index);                \
      |             ^~~~~~~~~~~~~
/scratch/sw/wdb2dc2/ncbi-vdb/2.11.2/ncbi-vdb-2.11.2/libs/klib/judy/JudyIns.c:1592:13: note: in expansion of macro ‘JU_IMMSETINPLACE’
 1592 |             JU_IMMSETINPLACE(5, uint8_t *, cJ1_JPIMMED_5_02, j__udySearchLeaf5,
      |             ^~~~~~~~~~~~~~~~

There are more similar lines on this. This one looks pretty ugly. Is it a false positive on buffer overflow?

installation does not include files required by sra-tools

sra-tools depends on headers and files in the ilib directory that are not included in the installation output.

See ncbi/sra-tools#11

hdf5 not recognized on Arch

Arch linux 5.3.6-arch1

hdf5-1.10.5-1
gcc/g++ 9.2.0
clang 9.0.0

Hello, I have libhdf5 shared objects, C++ files etc in /usr/lib and headers in /usr/include

>ls /usr/include/hdf5
hdf5.h hdf5_hl.h hdf5.mod
>ls /usr/include/libhdf5 # symlinks to the above
libhdf5.h libhdf5_hl.h libhdf5.mod

Any documentation?

How to create schema?
Any notes on folder structure?
How to define indexes?

Installed ncbi-vdb package cannot be found by sratools

I just installed ncbi-vdb 2.8.0. Everything was found in the ./configure step and installation completed without warning or error.
When I ran ./configure for sratools, however, no ncbi-vdb package was found.

I tried many things people shared online, mainly specifying locations in the ./configure step, but couldn't solve the issue. I started to think whether ncbi-vdb was properly installed. I checked the possible include directory, since ./configure for sratools showed that there was no ncbi-vdb "includes".

$ ls /usr/local/ncbi/ncbi-vdb/include/ncbi-vdb/
NGS.hpp

It is strange because there is only one file "NGS.hpp" found in the include directory. Can anyone tell me whether this is normal or whether the include files were properly installed?

	elseif ( ${CMAKE_HOST_SYSTEM_PROCESSOR} STREQUAL "arm64")
	set(ARCH "aarch64")
	elseif ( ${CMAKE_HOST_SYSTEM_PROCESSOR} STREQUAL "aarch64")
	set(ARCH "aarch64")

	if ( "GNU" STREQUAL "${CMAKE_C_COMPILER_ID}")
	include_directories(interfaces/cc/gcc)
	include_directories(interfaces/cc/gcc/${ARCH})

ncbi / ncbi-vdb Goto Github PK

ncbi-vdb's Introduction

The NCBI SRA ( Sequence Read Archive )

Contact:

Change Log

NCBI VDB

ANNOUNCEMENTS:

Affected repositories

Antivirus Warnings

ncbi-vdb's People

Contributors

Stargazers

Watchers

Forkers

ncbi-vdb's Issues

Recommend Projects

Recommend Topics

Recommend Org