malfoy / bwise Goto Github PK

BWISE has been blocked on btrim for ~10h. It has not produced the haploid assembly
OUTPUT:

python ~/BWISE26-04/BWISE/Bwise.py -x ANature_interleave.fasta -p 5 -S 5 -K 401 -H 1 -t 45 -o /media/urbe/MyADrive/ANatureAssembly26-04-18 

*** This is BWISE - High order De Bruijn graph assembler ***

Binaries are in: /home/urbe/BWISE26-04/BWISE/bin
The command line was: /home/urbe/BWISE26-04/BWISE/Bwise.py -x ANature_interleave.fasta -p 5 -S 5 -K 401 -H 1 -t 45 -o /media/urbe/MyADrive/ANatureAssembly26-04-18
[Warning] /media/urbe/MyADrive/ANatureAssembly26-04-18 directory already exists, BWISE will use it.
Results will be stored in:  /media/urbe/MyADrive/ANatureAssembly26-04-18

[11:14:00 26/04/2018] --> Starting Graph construction and Super Reads generation...

[...]

 The file contigs_k401.fa has been produced !


	/home/urbe/BWISE26-04/BWISE/bin/bgreat  -Z -g dbg401.fa -u dbg401.fa -i 1000 -o 0 -t 8 -k 401 -a 15

	/home/urbe/BWISE26-04/BWISE/bin/btrim -u popped_dbg.fa -k 401    -c 45 -h 8 -o crushed_dbg.fa

Hi, I was able to compile BWISE but can't run the tests all the way through. It fails at this step:

#Graph 3: Construction... 
#Current date & time Tue Jul 17 16:25:12 2018
	/home/bsiranos/software/BWISE/bin/bcalm -max-memory 10000 -in /data/bsiranos/software/BWISE/data/testFolder/contigs_k101.fa -kmer-size 102 -abundance-min 1 -out /data/bsiranos/software/BWISE/data/testFolder/out  -nb-cores 4

[ERROR] There was a problem generating "/data/bsiranos/software/BWISE/data/testFolder/out.unitigs.fa".
[FATAL ERROR] One or several files could not be generated.
To find out why, try `Bwise --help` and/or check the logs files of the various steps of the pipeline (logs/logBloocoo, logs/logBcalm, logs/logTips, logs/logBgreat, logs/logK2000).
IT WORKS !
N50: 1
L50: 1
N80: 1
L80: 2
N90: 1
L90: 2

Here's the tail end of testFolder/logs/logBcalm. Something related to hdf5?

debug: not deleting glue files
Finding links between unitigs                   16:25:10     memory [current, maxRSS]: [ 173,  173] MB 
step 1 pass 0                                   16:25:10     memory [current, maxRSS]: [ 173,  173] MB 
step 2 (0kmers/0extremities)                    16:25:10     memory [current, maxRSS]: [ 173,  173] MB 
step 1 pass 1                                   16:25:10     memory [current, maxRSS]: [ 173,  173] MB 
step 2 (0kmers/0extremities)                    16:25:10     memory [current, maxRSS]: [ 173,  173] MB 
step 1 pass 2                                   16:25:10     memory [current, maxRSS]: [ 173,  173] MB 
step 2 (0kmers/0extremities)                    16:25:10     memory [current, maxRSS]: [ 173,  173] MB 
step 1 pass 3                                   16:25:10     memory [current, maxRSS]: [ 173,  173] MB 
step 2 (0kmers/0extremities)                    16:25:10     memory [current, maxRSS]: [ 173,  173] MB 
step 1 pass 4                                   16:25:10     memory [current, maxRSS]: [ 173,  173] MB 
step 2 (1kmers/1extremities)                    16:25:10     memory [current, maxRSS]: [ 173,  173] MB 
step 1 pass 5                                   16:25:10     memory [current, maxRSS]: [ 173,  173] MB 
step 2 (1kmers/1extremities)                    16:25:10     memory [current, maxRSS]: [ 173,  173] MB 
step 1 pass 6                                   16:25:10     memory [current, maxRSS]: [ 173,  173] MB 
step 2 (0kmers/0extremities)                    16:25:10     memory [current, maxRSS]: [ 173,  173] MB 
step 1 pass 7                                   16:25:10     memory [current, maxRSS]: [ 173,  173] MB 
step 2 (0kmers/0extremities)                    16:25:10     memory [current, maxRSS]: [ 173,  173] MB 
gathering links from disk                       16:25:10     memory [current, maxRSS]: [ 173,  173] MB 
Done finding links between unitigs              16:25:10     memory [current, maxRSS]: [ 173,  173] MB 
BCALM 2, git commit e9ba83c
setting storage type to hdf5
HDF5-DIAG: Error detected in HDF5 (1.8.18) thread 0:
  #000: /home/bsiranos/software/BWISE/src/bcalm/gatb-core/gatb-core/thirdparty/hdf5/src/H5A.c line 407 in H5Aopen(): unable to load attribute info from object header for attribute: 'version'
    major: Attribute
    minor: Unable to initialize object
  #001: /home/bsiranos/software/BWISE/src/bcalm/gatb-core/gatb-core/thirdparty/hdf5/src/H5Oattribute.c line 537 in H5O_attr_open_by_name(): can't locate attribute: 'version'
    major: Attribute
    minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.8.18) thread 0:
  #000: /home/bsiranos/software/BWISE/src/bcalm/gatb-core/gatb-core/thirdparty/hdf5/src/H5A.c line 677 in H5Aget_space(): not an attribute
    major: Invalid arguments to routine
    minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.8.18) thread 0:
  #000: /home/bsiranos/software/BWISE/src/bcalm/gatb-core/gatb-core/thirdparty/hdf5/src/H5S.c line 883 in H5Sget_simple_extent_dims(): not a dataspace
    major: Invalid arguments to routine
    minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.8.18) thread 0:
  #000: /home/bsiranos/software/BWISE/src/bcalm/gatb-core/gatb-core/thirdparty/hdf5/src/H5A.c line 634 in H5Aread(): not an attribute
    major: Invalid arguments to routine
    minor: Inappropriate type
EXCEPTION: Unable to open bank '/data/bsiranos/software/BWISE/data/testFolder/contigs_k101.fa' (if it is a list of files, perhaps some of the files inside don't exist)

Hello,
I got an error with Bloocoo

/home/alessandro/BWISE/bin/Bloocoo32: /usr/local/bin/anaconda3/lib/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /home/alessandro/BWISE/bin/Bloocoo32)
/home/alessandro/BWISE/bin/Bloocoo32: /usr/local/bin/anaconda3/lib/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /home/alessandro/BWISE/bin/Bloocoo32)
/home/alessandro/BWISE/bin/Bloocoo32: /usr/local/bin/anaconda3/lib/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /home/alessandro/BWISE/bin/Bloocoo32)
Maybe this should be checked at the installation step? I don't think by default anaconda3 will be installed. 

I think anaconda3 and libstdc++ is not installed by default anyway, maybe it should be checked at install time or in a "prerequisite" section?

If I launch Bwise.py in e.g. $HOME/1/2/3/4/5 without specifying any output directory, the output folder gets set to $HOME/1/2/3/4: this is the folder where all temporary files are created, but BWISE looks for them in $HOME/1/2/3/4/5 and crashes.

Hi, I run BWISE (commit cac2d0) with minimal command bwise -x Sm2.trim.norm.180.all.fastq -t 32 and it crushed.

Std out:

2
Reads Correction
Graph construction 0
Read mapping on the graph 0
Graph construction 1
Read mapping on the graph 1
Graph construction 2
Read mapping on the graph 2
Graph construction 3
Read mapping on the graph 3
Graph construction 4
Read mapping on the graph 4
SuperReads Compactions
The end

Error log:

mkdir: cannot create directory `.': File exists
sh: line 1: 74736 Segmentation fault      (core dumped) /scratch/beegfs/monthly/kjaron/src/BWISE/bin/numbersFilter paths 2 2 > cleanedPaths 2>> logs/logBgreat
SR load super reads 
SR add reverse complements 0
sorting 0
remove inclusions 0
Traceback (most recent call last):
  File "/scratch/beegfs/monthly/kjaron/src/BWISE/bin/K2000.py", line 232, in <module>
    main()  
  File "/scratch/beegfs/monthly/kjaron/src/BWISE/bin/K2000.py", line 204, in main
    SR=remove_strict_inclusions(SR)
  File "/scratch/beegfs/monthly/kjaron/src/BWISE/bin/K2000.py", line 72, in remove_strict_inclusions
    sys.stderr.write("Removing inclusions, "+str(checked)+" checked. Size SR "+str(len(SR))+" %.2f"%(100*checked/len(SR))+"%\n")
ZeroDivisionError: division by zero
Print GFA Nodes
        100.00% -- 0 error(s)
Print GFA Edges
        100.00%

Hi, I'm facing an issue when compiling BWISE at step one itself, encountering the error.
I put binaries in /home/igib/visanu2/BWISE/bin PHASE ONE LAUNCHER: BWISE there was a problem with binary compilation, check logs
The OS I'm using is Ubuntu 22.04 LTS, CMAKE version 3.22.1, Make version 4.3 and GCC version 11.2.0 in a Python 3.7.13 environment.

BWISE seems to be working fine as long as k<220, then fails
Here is the output:
Reads Correction
Graph construction 0
Read mapping on the graph 0
Graph construction 1
Read mapping on the graph 1
Graph construction 2
Read mapping on the graph 2
Graph construction 3
Read mapping on the graph 3
Graph construction 4
Read mapping on the graph 4
Graph construction 5
Read mapping on the graph 5
sh: line 1: 56060 Killed /cluster/scratch4/ucbtlot/BWISEbin/bgreat -k 220 -x reads_corrected1.fa -u reads_corrected2.fa -g dbg5.fa -t 10 -c -m 0 -e 1 >> logs/logBgreat 2>> logs/logBgreat
Graph construction 6
h5dump error: unable to get link info from "histogram/histogram"
Read mapping on the graph 6
Graph construction 7
h5dump error: unable to get link info from "histogram/histogram"
Read mapping on the graph 7
Graph construction 8
h5dump error: unable to get link info from "histogram/histogram"
Read mapping on the graph 8
Graph construction 9
h5dump error: unable to get link info from "histogram/histogram"
Read mapping on the graph 9
Graph construction 10
h5dump error: unable to get link info from "histogram/histogram"
Read mapping on the graph 10
Graph construction 11
h5dump error: unable to get link info from "histogram/histogram"
Read mapping on the graph 11
Graph construction 12
h5dump error: unable to get link info from "histogram/histogram"
Read mapping on the graph 12

This should allow us to obtain a better level of continuity when assembling haploid genome (or diploid genomes with a very low level of heterozygosity).

Hello,
running Bwise returns this error

/home/alessandro/BWISE/Bwise.py -x reads.fa -t 2 -o test

*** This is BWISE - High order De Bruijn graph assembler ***

Binaries are in: /home/alessandro/BWISE/bin
The command line was: /home/alessandro/BWISE/Bwise.py -x reads.fa -t 2 -o test
Results will be stored in:  /home/alessandro/Documents/PhD/Bwise_test/test

[14:48:48 18/09/2017] --> Starting Read Correction with Bloocoo...
	Correction step 1
		/home/alessandro/BWISE/bin/Bloocoo32 -file /home/alessandro/Documents/PhD/Bwise_test/reads.fa  -slow -kmer-size 31 -nbits-bloom 24  -out reads_corrected1.fa -nb-cores 2
			ln -fs /home/alessandro/Documents/PhD/Bwise_test/test/reads_corrected1.fa  /home/alessandro/Documents/PhD/Bwise_test/test/reads_corrected.fa

[15:42:37 18/09/2017] --> Correction Done
Correction took:  0:53:49

[15:42:37 18/09/2017] --> Starting Graph construction and Super Reads generation...
Unexpected error during graph construction: <class 'FileNotFoundError'>
[FATAL ERROR] 
Try `Bwise --help` for more information
zsh: exit 1     /home/alessandro/BWISE/Bwise.py -x reads.fa -t 2 -o test

I installed Bwise today, 18th of September 2017, so it's the very last version.

Any idea? the input file is in correct fasta format.

Hi!

I plan to try your assembler for my data which are composed of 3 PE libraries:

• 180
• 350
• 550

I am asking your help how to use your assembler for this kind of data.

1. The assembler can handle multiple libraries ?

2. If yes, it seems that I have to use the -x option: Does it has to be formated in interleaved format ?

3. Can I use -x 3 times or have to merge the 3 libraries.

Thanks for your support.

Hi,

Do you have some statistics regarding the running time of the assembler with how many CPU associated ?

Hi,

I have run the assembly and apparently it works without error (see log below).
But I would like to know what is the name of the final file (the contigs) ?

`
./Bwise.py -x Sm2.trim.180.pair12.fastq -k 300 -t 30 -o out

• This is BWISE - High order De Bruijn graph assembler *

Binaries are in: /scratch/local/monthly/ptranvan/BWISE/bin
The command line was: ./Bwise.py -x Sm2.trim.180.pair12.fastq -k 300 -t 30 -o out
Results will be stored in: /scratch/local/monthly/ptranvan/BWISE/out

[08:49:30 20/04/2017] --> Starting Read Correction with Bloocoo...
Correction step 1
/scratch/local/monthly/ptranvan/BWISE/bin/Bloocoo32 -file /scratch/local/monthly/ptranvan/BWISE/Sm2.trim.180.pair12.fastq -slow -kmer
-size 31 -nbits-bloom 24 -out reads_corrected1.fa -nb-cores 30
Correction step 2
/scratch/local/monthly/ptranvan/BWISE/bin/Bloocoo64 -file /scratch/local/monthly/ptranvan/BWISE/out/reads_corrected1.fa -slow -kmer-s
ize 63 -nbits-bloom 24 -out reads_corrected2.fa -nb-cores 30
Correction step 3
/scratch/local/monthly/ptranvan/BWISE/bin/Bloocoo128 -file /scratch/local/monthly/ptranvan/BWISE/out/reads_corrected2.fa -slow -kmer-
size 95 -nbits-bloom 24 -out reads_corrected3.fa -nb-cores 30
Correction step 4
/scratch/local/monthly/ptranvan/BWISE/bin/Bloocoo128 -file /scratch/local/monthly/ptranvan/BWISE/out/reads_corrected3.fa -slow -kmer-size 127 -nbits-bloom 24 -out reads_corrected4.fa -nb-cores 30
ln -fs /scratch/local/monthly/ptranvan/BWISE/out/reads_corrected4.fa /scratch/local/monthly/ptranvan/BWISE/out/reads_corrected.fa

[13:35:31 20/04/2017] --> Correction Done
Correction took: 4:46:01

[13:35:31 20/04/2017] --> Starting Graph construction and Super Reads generation...
#Graph 1: Construction...
/scratch/local/monthly/ptranvan/BWISE/bin/bcalm -in /scratch/local/monthly/ptranvan/BWISE/out/bankBcalm.txt -kmer-size 51 -abundance-min 2 -out /scratch/local/monthly/ptranvan/BWISE/out/out -nb-cores 30
#Cleaning...
/scratch/local/monthly/ptranvan/BWISE/bin/kMILL out.unitigs.fa 50 49
/scratch/local/monthly/ptranvan/BWISE/bin/tipCleaner out_out.unitigs.fa.fa 50 101
/scratch/local/monthly/ptranvan/BWISE/bin/kMILL tiped.fa 50 49
mv out_tiped.fa.fa dbg51.fa
#Read mapping with BGREAT...
/scratch/local/monthly/ptranvan/BWISE/bin/bgreat -M -k 51 -i 10 -x reads_corrected.fa -g dbg51.fa -t 30 -a 63 -m 0 -e 100
/scratch/local/monthly/ptranvan/BWISE/bin/numbersFilter paths 2 cleanedPaths_51 2 dbg51.fa 51
/scratch/local/monthly/ptranvan/BWISE/bin/run_K2000.sh cleanedPaths_51 dbg51.fa 51 compacted_unitigs_k51.gfa compacted_unitigs_k51.fa
#Graph 2: Construction...
/scratch/local/monthly/ptranvan/BWISE/bin/bcalm -in /scratch/local/monthly/ptranvan/BWISE/out/compacted_unitigs_k51.fa -kmer-size 101 -abundance-min 1 -out /scratch/local/monthly/ptranvan/BWISE/out/out -nb-cores 30
#Cleaning...
/scratch/local/monthly/ptranvan/BWISE/bin/kMILL out.unitigs.fa 100 99
/scratch/local/monthly/ptranvan/BWISE/bin/tipCleaner out_out.unitigs.fa.fa 100 151
/scratch/local/monthly/ptranvan/BWISE/bin/kMILL tiped.fa 100 99
mv out_tiped.fa.fa dbg101.fa
#Read mapping with BGREAT...
/scratch/local/monthly/ptranvan/BWISE/bin/bgreat -M -k 101 -i 10 -x reads_corrected.fa -g dbg101.fa -t 30 -a 63 -m 0 -e 100
/scratch/local/monthly/ptranvan/BWISE/bin/numbersFilter paths 2 cleanedPaths_101 2 dbg101.fa 101
/scratch/local/monthly/ptranvan/BWISE/bin/run_K2000.sh cleanedPaths_101 dbg101.fa 101 compacted_unitigs_k101.gfa compacted_unitigs_k101.fa
#Graph 3: Construction...
/scratch/local/monthly/ptranvan/BWISE/bin/bcalm -in /scratch/local/monthly/ptranvan/BWISE/out/compacted_unitigs_k101.fa -kmer-size 151 -abundance-min 1 -out /scratch/local/monthly/ptranvan/BWISE/out/out -nb-cores 30
#Cleaning...
/scratch/local/monthly/ptranvan/BWISE/bin/kMILL out.unitigs.fa 150 149
/scratch/local/monthly/ptranvan/BWISE/bin/tipCleaner out_out.unitigs.fa.fa 150 201
/scratch/local/monthly/ptranvan/BWISE/bin/kMILL tiped.fa 150 149
mv out_tiped.fa.fa dbg151.fa
#Read mapping with BGREAT...
/scratch/local/monthly/ptranvan/BWISE/bin/bgreat -M -k 151 -i 10 -x reads_corrected.fa -g dbg151.fa -t 30 -a 63 -m 0 -e 100
/scratch/local/monthly/ptranvan/BWISE/bin/numbersFilter paths 2 cleanedPaths_151 2 dbg151.fa 151
/scratch/local/monthly/ptranvan/BWISE/bin/run_K2000.sh cleanedPaths_151 dbg151.fa 151 compacted_unitigs_k151.gfa compacted_unitigs_k151.fa
#Graph 4: Construction...
/scratch/local/monthly/ptranvan/BWISE/bin/bcalm -in /scratch/local/monthly/ptranvan/BWISE/out/compacted_unitigs_k151.fa -kmer-size 201 -abundance-min 1 -out /scratch/local/monthly/ptranvan/BWISE/out/out -nb-cores 30
#Cleaning...
/scratch/local/monthly/ptranvan/BWISE/bin/kMILL out.unitigs.fa 200 199
/scratch/local/monthly/ptranvan/BWISE/bin/tipCleaner out_out.unitigs.fa.fa 200 251
/scratch/local/monthly/ptranvan/BWISE/bin/kMILL tiped.fa 200 199
mv out_tiped.fa.fa dbg201.fa
#Read mapping with BGREAT...
/scratch/local/monthly/ptranvan/BWISE/bin/bgreat -M -k 201 -i 10 -x reads_corrected.fa -g dbg201.fa -t 30 -a 63 -m 0 -e 100
/scratch/local/monthly/ptranvan/BWISE/bin/numbersFilter paths 2 cleanedPaths_201 2 dbg201.fa 201
/scratch/local/monthly/ptranvan/BWISE/bin/run_K2000.sh cleanedPaths_201 dbg201.fa 201 compacted_unitigs_k201.gfa compacted_unitigs_k201.fa
#Graph 5: Construction...
/scratch/local/monthly/ptranvan/BWISE/bin/bcalm -in /scratch/local/monthly/ptranvan/BWISE/out/compacted_unitigs_k201.fa -kmer-size 251 -abundance-min 1 -out /scratch/local/monthly/ptranvan/BWISE/out/out -nb-cores 30
#Cleaning...
/scratch/local/monthly/ptranvan/BWISE/bin/kMILL out.unitigs.fa 250 249
/scratch/local/monthly/ptranvan/BWISE/bin/tipCleaner out_out.unitigs.fa.fa 250 301
/scratch/local/monthly/ptranvan/BWISE/bin/kMILL tiped.fa 250 249
mv out_tiped.fa.fa dbg251.fa
#Read mapping with BGREAT...
/scratch/local/monthly/ptranvan/BWISE/bin/bgreat -M -k 251 -i 10 -x reads_corrected.fa -g dbg251.fa -t 30 -a 63 -m 0 -e 100
/scratch/local/monthly/ptranvan/BWISE/bin/numbersFilter paths 2 cleanedPaths_251 2 dbg251.fa 251
/scratch/local/monthly/ptranvan/BWISE/bin/run_K2000.sh cleanedPaths_251 dbg251.fa 251 compacted_unitigs_k251.gfa compacted_unitigs_k251.fa
[08:34:52 25/04/2017] --> Done!
Graph Construction took: 114:59:20

The end !
BWISE assembly took: 119:45:22
`

GPL licenses mandate that a copy of the license be distributed with source and binaries. We need that file for Bioconda to comply with the license restrictions.

what is the difference between compacted_unitigs*.fa and dbg*.fa

Bwise terminated at Graph 1: Construction steps. All the log files are empty !
Bwise version used : https://anaconda.org/bioconda/bwise

`(base) jit@jit-HP-Pro-3335-MT:~/Downloads/setu$ bwise -x interleaved.fa -o bwise_assembly

*** This is BWISE - High order De Bruijn graph assembler ***

The command line was: /home/jit/anaconda3/bin/bwise -x interleaved.fa -o bwise_assembly
[Warning] bwise_assembly directory already exists, BWISE will use it.
Results will be stored in: /home/jit/Downloads/setu/bwise_assembly

[17:37:32 16/05/2020] --> Starting Graph construction and Super Reads generation...
#Graph 1: Construction...
#Current date & time Sat May 16 17:37:32 2020
bcalm -max-memory 10000 -in /home/jit/Downloads/setu/bwise_assembly/bankBcalm.txt -kmer-size 63 -abundance-min 3 -out /home/jit/Downloads/setu/bwise_assembly/out -nb-cores 20

[ERROR] There was a problem generating "/home/jit/Downloads/setu/bwise_assembly/out.unitigs.fa".
[FATAL ERROR] One or several files could not be generated.
To find out why, try Bwise --help and/or check the logs files of the various steps of the pipeline (logs/logBloocoo, logs/logBcalm, logs/logTips, logs/logBgreat, logs/logK2000).
`

In the "stable" branch, we should specify the exact version for each dependant (to prevent that dependency updates break the pipeline); whereas the "dev" branch should use the latest version of each dependency.