bcgsc / goldrush Goto Github PK

View Code? Open in Web Editor NEW

30.0 10.0 1.0 4.34 MB

Linear-time de novo Long Read Assembler

License: GNU General Public License v3.0

C++ 88.17% Makefile 9.46% Shell 1.17% Meson 1.20%

bioinformatics bioinformatics-tool genome-assembly long-read-sequencing

goldrush's People

Contributors

Stargazers

Watchers

Forkers

oneillkza

goldrush's Issues

Consolidate `T` and `G` parameters

Currently, we have two different parameters for specifying genome size: T and G
Add a parser for the goldrush-path code to accept genome sizes in formats like 3e6, like Tigmint-long can, and then consolidate these

Using existing Draft Assembly in place of Gold Path Input

Hello,
I'm wondering if its possible to leverage the editing and polishing steps of goldrush on an existing draft assembly without running the goldpath portion?
Thank you,
Stuart

Clean-up makefile help page

Remove extraneous targets, parameters that we can reasonably expect the user won't change

Combine silver path and golden path generation in one step

Silver path and golden path can be combined in one step to avoid regenerating a bloom filter and allow easy propagation of variables without interacting with the makefile

automate m and phred parameters

miller required for goldrush_test_demo.sh

Hello! 👋🏼

I installed goldrush with conda and wanted to test it out by using goldrush_test_demo.sh; the tests seem to run fine but the script itself will end in an error due to running mlr on line 12:

make[1]: Leaving directory '/lustre/fs5/vgl/scratch/labueg/2022biohackathon/goldrush/goldrush/tests'
ln -sf goldrush_test_golden_path.goldrush-edit-polished.span2.dist500.tigmint.fa.k40.w250.z1000.ntLink.gap_fill.5rounds.fa goldrush_test_golden_path.goldrush-edit-polished.span2.dist500.tigmint.fa.k40.w250.ntLink-5rounds.fa
echo "Done GoldRush-Path + GoldRush-Edit + Tigmint-long + 5 ntLink rounds! Your final assembly can be found in: goldrush_test_golden_path.goldrush-edit-polished.span2.dist500.tigmint.fa.k40.w250.ntLink-5rounds.fa"
Done GoldRush-Path + GoldRush-Edit + Tigmint-long + 5 ntLink rounds! Your final assembly can be found in: goldrush_test_golden_path.goldrush-edit-polished.span2.dist500.tigmint.fa.k40.w250.ntLink-5rounds.fa
./goldrush_test_demo.sh: line 12: mlr: command not found

The error doesn't happen after I install miller and run goldrush_test_demo.sh again, here is the end of that stdout:

Done ntLink rounds!  Final scaffolds found in goldrush_test_golden_path.goldrush-edit-polished.span2.dist500.tigmint.fa.k40.w250.z1000.ntLink.5rounds.fa
make[1]: Leaving directory '/lustre/fs5/vgl/scratch/labueg/2022biohackathon/goldrush/goldrush/tests'
ntLink_rounds clean target=goldrush_test_golden_path.goldrush-edit-polished.span2.dist500.tigmint.fa t=4 k=40 w=250 z=1000 rounds=5 reads=test_reads.fq
make[1]: Entering directory '/lustre/fs5/vgl/scratch/labueg/2022biohackathon/goldrush/goldrush/tests'
rm -f goldrush_test_golden_path.goldrush-edit-polished.span2.dist500.tigmint.fa.k40.w250.z1000.*verbose_mapping.tsv
make[1]: Leaving directory '/lustre/fs5/vgl/scratch/labueg/2022biohackathon/goldrush/goldrush/tests'
ln -sf goldrush_test_golden_path.goldrush-edit-polished.span2.dist500.tigmint.fa.k40.w250.z1000.ntLink.gap_fill.5rounds.fa goldrush_test_golden_path.goldrush-edit-polished.span2.dist500.tigmint.fa.k40.w250.ntLink-5rounds.fa
echo "Done GoldRush-Path + GoldRush-Edit + Tigmint-long + 5 ntLink rounds! Your final assembly can be found in: goldrush_test_golden_path.goldrush-edit-polished.span2.dist500.tigmint.fa.k40.w250.ntLink-5rounds.fa"
Done GoldRush-Path + GoldRush-Edit + Tigmint-long + 5 ntLink rounds! Your final assembly can be found in: goldrush_test_golden_path.goldrush-edit-polished.span2.dist500.tigmint.fa.k40.w250.ntLink-5rounds.fa

Test successful!

Just wanted to bring this up as miller isn't listed as one of the requirements for the tool. Thanks!

Can goldrush be used to assemble hifi-reads?

Hi,

Can I use goldrush to assemble hifi-reads?

Best,
Kun

Goldrush for PacBio or ONT + PacBio

Hi!

I was wondering if Goldrush has been tested / is used / is recommended for PacBio data - or ONT + PacBio data? If so, are there any specific recommendations? I have data sets for two species with around human-sized genomes. One with 40x ONT + 7x PacBio and another with 10x ONT + 60x PacBio - all data generated around 2020.

Thank you :)
Eric

Error code 11 when running GoldRush-Path on Rice

The command works fine when using P=10 but when I switch to P=15, I consistently get the same error. This is for v1.0.0 using Lauren's installation.

GoldRush Command:
nohup /usr/bin/time -pv goldrush run reads=osativa.ont G=373e6 P=15 a=1 m=20000 track_time=1 polisher_mapper=minimap2 span=2 dist=500 k_ntLink=32 w_ntLink=75 polisher=goldrush-edit &> goldrush.log &

Log:

nohup: ignoring input
command time -v -o w16_x10_silver_path_5.fq.time goldrush-path  -k 22 -w 16 -t 1000 -u 5 -a 1 -o 0.1 -p w16_x10_silver_path -i osativa.ont.fq  -h 3 -j 48 -x10 -P 15 -d 5 -s 1011011110110111101101 -g 373e6 -b 10 -r 0.9 --silver_path -M 5 -m 20000
Using preset spaced seed
with:
	span: 22
	weight: 16
Calculating 5 silver path(s)
Using:
	tile length: 1000
	block size: 10
	seed patterns: 3
	threshold: 10
	base seed pattern: 1011011110110111101101
	minimum unassigned tiles: 5
	maximum assigned tiles: 1
	expected hash space: 1119000064
	minimum average phred quality score: 15
	maximum average phred delta between first and second half of read: 5
	occupancy: 0.1
	jobs: 48
allocating bit vector
m_filterSize: 10620677568
finished allocating bit vector
in 0.5639
opening: osativa.ont.fq
inserting bit vector
finished inserting bit vector
in 41.2171
make: *** [/projects/btl/lcoombe/miniconda3/envs/goldrush_paper/bin/goldrush.make:201: w16_x10_silver_path_5.fq] Error 11
Command exited with non-zero status 2
	Command being timed: "goldrush run reads=osativa.ont G=373e6 track_time=1 polisher_mapper=minimap2 k_ntLink=32 w_ntLink=75 polisher=goldrush-edit"
	User time (seconds): 70.47
	System time (seconds): 38.00
	Percent of CPU this job got: 202%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:53.60
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 2857844
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 956765
	Voluntary context switches: 4417858
	Involuntary context switches: 129
	Swaps: 0
	File system inputs: 0
	File system outputs: 32
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 2

Add installation instructions with tarball to readme

Consolidate j, t parameters

Update goldrush default settings

Update k/w ntLink defaults

Add dependencies to README

Transient seg fault when running GoldRush on mac

Have seen (through CI and locally) a transient segmentation fault popping up when running GoldRush on macOS:

opening: test_reads.fq
inserting bit vector
make: *** [goldrush_test_silver_path_5.fq] Segmentation fault: 11

Above is snippet from failed CI on macOS - when check is re-run, it generally will eventually pass OK.

Add CI

set-up CI for the repo: testing compiling GoldRush-path, running test data, etc.

Fix white space, hardcoded parameters in goldrush makefile

Make white space consistent, ensure consistency of thread parameters (no hardcoding, etc)

Add `ulimit -u` command in driver makefile

Multiple users have reported GoldRush-Edit issues (Ex. #95 ) that appear to be related to the max user processes settings on their machine
Include command ulimit -u 256000 to increase the limit and avoid these errors

zsh & pipefail

I am getting stuck at the tigmint step with the following error:
tigmint-make tigmint-long draft=w16_x10_golden_path.goldrush-edit-polished reads=long_reads cut=250 t=32 G=94000000 span=2 dist=500
zsh: no such option: pipefail
zsh: no such option: pipefail
zsh: no such option: pipefail
zsh: no such option: pipefail
make[1]: Entering directory /.../.../.../.../goldrush' gzip -dc | /long-to-linked -l250 -m2000 -g94000000 | gzip > long_reads.cut250.fa.gz zsh: no such option: pipefail make[1]: *** [long_reads.cut250.fa.gz] Error 1 make[1]: Leaving directory /.../.../.../.../goldrush'
make: *** [/.../.../.../.../conda/envs/goldrush/bin/goldrush.make:246: w16_x10_golden_path.goldrush-edit-polished.cut250.tigmint.fa] Error 2

add block size parameter to makefile

bc unknown command

Hello i am trying to use goldrush assembler on my diploid genome of mosquito and when i launch the pipeline every works fine but for one step when it launch : goldrush-edit-ntedit script i have an error because he doesn"t know the bc commande i think its at this line :

size_diff=$(echo "scale=4; $output_size / $input_size" | bc).

But it is very strange because when i am on my terminal i can use bc command but when i launch the pipeline, after he fails i can't use anymore bc command.

I have installed it well with conda/mamba.

Thanks for your answers.
Théo

terminate called after throwing an instance of 'std::system_error'

I have installed goldrush from source (and all required dependencies) and I am trying to run goldrush_test_demo.sh, but I am getting the following errors:

./goldrush_test_demo.sh

...
Launching GoldRush
make: command: Command not found
make: command: Command not found
goldrush-path  -k 22 -w 16 -t 1000 -u 5 -a 1 -o 0.1 -p goldrush_test_silver_path -i test_reads.fq  -h 3 -j 4 -x10 -P 15 -d 5 -s 1011011110110111101101 -g 1e6 -b 10 -r 0.9 --silver_path -M 5 -m 20000
Using preset spaced seed
...

I have make installed and it is in the PATH, so I am not sure what is causing that

$ which make
/usr/bin/make
$ make -v
GNU Make 4.2.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2016 Free Software Foundation, Inc.
Licence GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Then, the run continues, but only the first 2 stages (GoldRush-Path and GoldRush-Edit) are completed

...
[2022-12-10 17:48:52][INFO] Polisher done
echo "Done GoldRush-Path + GoldRush-Edit! GoldRush-Edit polished golden path can be found in: goldrush_test_golden_path.goldrush-edit-polished.fa"
Done GoldRush-Path + GoldRush-Edit! GoldRush-Edit polished golden path can be found in: goldrush_test_golden_path.goldrush-edit-polished.fa
tigmint-make tigmint-long draft=goldrush_test_golden_path.goldrush-edit-polished reads=test_reads cut=250 t=4 G=1e6 span=2 dist=500
make[1]: Entering directory '/home/duda5/soft/goldrush/tests'
sh -c 'gunzip -c  | \
/home/linuxbrew/.linuxbrew/Cellar/tigmint/1.2.5/libexec/bin/tigmint_estimate_dist.py - -n 1000000 -o test_reads.tigmint-long.params.tsv'
gzip: compressed data not read from a terminal. Use -f to force decompression.
For help, type: gzip -h
Traceback (most recent call last):
  File "/home/linuxbrew/.linuxbrew/Cellar/tigmint/1.2.5/libexec/bin/tigmint_estimate_dist.py", line 63, in <module>
    main()
  File "/home/linuxbrew/.linuxbrew/Cellar/tigmint/1.2.5/libexec/bin/tigmint_estimate_dist.py", line 55, in main
    dist = int(np.percentile(read_lengths, args.p))
  File "<__array_function__ internals>", line 180, in percentile
  File "/home/duda5/anaconda3/lib/python3.9/site-packages/numpy/lib/function_base.py", line 4134, in percentile
    return _quantile_unchecked(
  File "/home/duda5/anaconda3/lib/python3.9/site-packages/numpy/lib/function_base.py", line 4383, in _quantile_unchecked
    r, k = _ureduce(a,
  File "/home/duda5/anaconda3/lib/python3.9/site-packages/numpy/lib/function_base.py", line 3702, in _ureduce
    r = func(a, **kwargs)
  File "/home/duda5/anaconda3/lib/python3.9/site-packages/numpy/lib/function_base.py", line 4552, in _quantile_ureduce_func
    result = _quantile(arr,
  File "/home/duda5/anaconda3/lib/python3.9/site-packages/numpy/lib/function_base.py", line 4658, in _quantile
    take(arr, indices=-1, axis=DATA_AXIS)
  File "<__array_function__ internals>", line 180, in take
  File "/home/duda5/anaconda3/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 190, in take
    return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode)
  File "/home/duda5/anaconda3/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 57, in _wrapfunc
    return bound(*args, **kwds)
IndexError: cannot do a non-empty take from an empty axes.
make[1]: *** [/home/linuxbrew/.linuxbrew/Cellar/tigmint/1.2.5/libexec/bin/tigmint-make:248: test_reads.tigmint-long.params.tsv] Error 1
make[1]: Leaving directory '/home/duda5/soft/goldrush/tests'
make: *** [/home/duda5/soft/goldrush_bin/bin/goldrush:250: goldrush_test_golden_path.goldrush-edit-polished.cut250.tigmint.fa] Error 2

When I try running goldrush on real data, it fails immediately and generates empty files

$ goldrush run -t 18 G=3e9 reads=ONT_pcd_treads.fastq
make: command: Command not found
make: command: Command not found
touch w16_x10_silver_path_5.fq
touch w16_x10_silver_path_all.fq
touch w16_x10_golden_path.fa
touch w16_x10_golden_path.goldrush-edit-polished.fa
touch w16_x10_golden_path.goldrush-edit-polished.cut250.tigmint.fa
touch w16_x10_golden_path.goldrush-edit-polished.span2.dist500.tigmint.fa
touch w16_x10_golden_path.goldrush-edit-polished.span2.dist500.tigmint.fa.k40.w250.z1000.ntLink.gap_fill.5rounds.fa
touch w16_x10_golden_path.goldrush-edit-polished.span2.dist500.tigmint.fa.k40.w250.ntLink-5rounds.fa

install instructions from release

tmpdir/shared_dir pass through from parent goldrush?

I'm running into an issue where I can run the conda installedgoldrush pipeline when in interactive mode without issue. However, when I launch on our AWS/HPC equivlement it fails with "device out of memory" error -- My current hypothesis this is due to goldrush-edit -m option which defaults to /dev/shm -- this doesn't exist on our job system because we don't have swap. But it doesn't seem like I can pass a tmpdir/shared location through the parent goldrush script?

My work around was to edit the goldrush-edit python script to default to writing args.shared_mem to /tmp/goldrush-{uuid.uuid4()}

optimize parameters for large genome

I'm interested in using Goldrush to assemble ONT reads of a large, highly repetitive diploid plant genome (haploid size~22 Gb). The long reads have been error-corrected using FMLRC2 based on an FM-index of Illumina short-read data from the same individual. Are the default parameters for Goldrush likely to be optimal for a genome of this size and complexity? If not, can you suggest changes that might improve them? I have a job running now that completed the silver path and golden path steps, and has been aligning the long reads to the golden path for weeks. The job is using 60 threads with minimap2, and is slowly making progress, so it is not hung, but it is much slower than other jobs aligning long reads to draft assemblies of the same genome. I'm wondering if this is a consequence of artifacts in the golden path due to inappropriate parameter settings.

Improve README documentation

Add information about dependencies, installation, parameters, running tips, user-set prefix, etc.

large number of 1bp or small contigs

I'm trying to assembly a ~2.3 Gbp genome from ~60X ONT reads (latest chemistry and Dorado basecalls). I get a very large number of contigs (40K) but many are 1bp or otherwise quite short, although contig N50 is relatively good (6Mbp) and contig N80 is ~3Mbp. This falls off right after with N95 around 35Kbp compared to 300-500Kbp N95 for other assemblers that otherwise behave similarly at the top end (raven and nextdenovo).
If I filter out contigs shorter than 2Kbp there are still 25K contigs. For the most part these contigs should not even exist, as the read set itself has no reads below 2Kbp.
Thanks, trying to get a handle on this output, seems like a very good assembler.

Error when running goldrush_test_demo

I've just started testing your program and ran into an issue when trying the demonstration.

I installed the software using conda as described and downloaded the github repository to find the demo-program.

When I ran the program (using slurm) I got the following error:

<...>
echo "Done GoldRush-Path! Golden path can be found in: goldrush_test_golden_path.fa"
Done GoldRush-Path! Golden path can be found in: goldrush_test_golden_path.fa
rm -f goldrush_test_silver_path_*.fq
Clean Done
goldrush-ulimit goldrush-edit --minimap2 -t4 -m/dev/shm goldrush_test_golden_path.fa test_reads.fq goldrush_test_golden_path.goldrush-edit-polished.fa
Running with 1030656 max processes: goldrush-edit --minimap2 -t4 -m/dev/shm goldrush_test_golden_path.fa test_reads.fq goldrush_test_golden_path.goldrush-edit-polished.fa
  File "/mnt/users/harag/.conda/envs/goldrush/bin/goldrush-edit", line 308
    while record := reader.read():
                  ^
SyntaxError: invalid syntax
make: *** [/mnt/users/harag/.conda/envs/goldrush/bin/goldrush.make:233: goldrush_test_golden_path.goldrush-edit-polished.fa] Error 1

The program had generated one file: goldrush_test_golden_path.fa. It's not quite the same name as in the README description, but is this the expected final output?

Can you tell what went wrong here?

-Harald

core dump in goldrush-edit

Hi, thanks for the great software!
I've installed goldrush through conda and now am trying to test goldrush on my system, but keep getting core dumps at the edit stage.
Here's the command and STDOUT. I've replaced some of the paths with ${HOME}.

$ ./goldrush_test_demo.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 97.4M  100 97.4M    0     0  1195k      0  0:01:23  0:01:23 --:--:-- 1213k
Launching GoldRush
goldrush-path  -k 22 -w 16 -t 1000 -u 5 -a 1 -o 0.1 -p goldrush_test_silver_path -i test_reads.fq  -h 3 -j 4 -x10 -P 15 -d 5 -s 1011011110110111101101 -g 1e6 -b 10 -r 0.9 --silver_path -M 5 -m 20000
Using preset spaced seed
with:
	span: 22
	weight: 16
Calculating 5 silver path(s)
Using:
	tile length: 1000
	block size: 10
	seed patterns: 3
	threshold: 10
	base seed pattern: 1011011110110111101101
	minimum unassigned tiles: 5
	maximum assigned tiles: 1
	expected hash space: 3000000
	minimum average phred quality score: 15
	maximum average phred delta between first and second half of read: 5
	occupancy: 0.1
	jobs: 4
allocating bit vector
m_filterSize: 28473728
finished allocating bit vector
in 0.0021
opening: test_reads.fq
inserting bit vector
finished inserting bit vector
in 11.8782
assigning tiles
assigned
in 17.8826
cat goldrush_test_silver_path_*.fq > goldrush_test_silver_path_all.fq
goldrush-path  -k 22 -w 16 -t 1000 -u 5 -a 1 -o 0.1 -p goldrush_test_golden_path -i goldrush_test_silver_path_all.fq -h 3 -j 4 -P 15 -d 5 -x10 -s 1011011110110111101101 -g 1e6 -b 10  -m 0
Using preset spaced seed
with:
	span: 22
	weight: 16
Calculating the golden path
Using:
	tile length: 1000
	block size: 10
	seed patterns: 3
	threshold: 10
	base seed pattern: 1011011110110111101101
	minimum unassigned tiles: 5
	maximum assigned tiles: 1
	expected hash space: 3000000
	minimum average phred quality score: 15
	maximum average phred delta between first and second half of read: 5
	occupancy: 0.1
	jobs: 4
allocating bit vector
m_filterSize: 28473728
finished allocating bit vector
in 0.0021
opening: goldrush_test_silver_path_all.fq
inserting bit vector
finished inserting bit vector
in 0.8096
assigning tiles
assigned
in 1.8664
echo "Done GoldRush-Path! Golden path can be found in: goldrush_test_golden_path.fa"
Done GoldRush-Path! Golden path can be found in: goldrush_test_golden_path.fa
rm -f goldrush_test_silver_path_*.fq
Clean Done
goldrush-edit --minimap2 -t4 goldrush_test_golden_path.fa test_reads.fq goldrush_test_golden_path.goldrush-edit-polished.fa
[2023-03-07 17:21:54][INFO] Building indexes and mappings...
[2023-03-07 17:22:01][INFO] make[1]: ディレクトリ `${HOME}/goldrush/tests' に入ります
goldrush-edit-index goldrush_test_golden_path.fa goldrush_test_golden_path.fa.index
minimap2 -t4 ${HOME}/goldrush/tests/goldrush_test_golden_path.fa ${HOME}/goldrush/tests/test_reads.fq >goldrush_test_golden_path.fa.test_reads.fq.paf
goldrush-edit-index test_reads.fq test_reads.fq.index
make[1]: ディレクトリ `${HOME}/goldrush/tests' から出ます
[2023-03-07 17:21:54][INFO] SeqIndex::SeqIndex: Building index for goldrush_test_golden_path.fa...
[2023-03-07 17:21:54][INFO] SeqIndex::SeqIndex: Done.
[2023-03-07 17:21:54][INFO] SeqIndex::save: Saving index to goldrush_test_golden_path.fa.index...
[2023-03-07 17:21:54][INFO] SeqIndex::save: Done.
[M::mm_idx_gen::0.041*1.06] collected minimizers
[M::mm_idx_gen::0.065*1.04] sorted minimizers
[M::main::0.065*1.04] loaded/built the index for 152 target sequence(s)
[M::mm_mapopt_update::0.070*1.04] mid_occ = 10
[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 152
[M::mm_idx_stat::0.072*1.03] distinct minimizers: 204938 (96.03% are singletons); average occurrences: 1.043; average spacing: 5.341; total length: 1141569
[M::worker_pipeline::6.764*1.00] mapped 3795 sequences
[M::main] Version: 2.24-r1122
[M::main] CMD: minimap2 -t4 ${HOME}/goldrush/tests/goldrush_test_golden_path.fa ${HOME}/goldrush/tests/test_reads.fq
[M::main] Real time: 6.773 sec; CPU: 6.780 sec; Peak RSS: 0.075 GB
[2023-03-07 17:22:00][INFO] SeqIndex::SeqIndex: Building index for test_reads.fq...
[2023-03-07 17:22:01][INFO] SeqIndex::SeqIndex: Done.
[2023-03-07 17:22:01][INFO] SeqIndex::save: Saving index to test_reads.fq.index...
[2023-03-07 17:22:01][INFO] SeqIndex::save: Done.

[2023-03-07 17:22:01][INFO] Indexes and mappings built.
[2023-03-07 17:22:01][INFO] Subsampling mapped reads to 40
[2023-03-07 17:22:01][INFO] SeqIndex::SeqIndex: Loading index from ${HOME}/goldrush/tests/goldrush_test_golden_path.fa.index...
[2023-03-07 17:22:01][INFO] SeqIndex::SeqIndex: Done!
[2023-03-07 17:22:01][INFO] SeqIndex::SeqIndex: Loading index from ${HOME}/goldrush/tests/test_reads.fq.index...
[2023-03-07 17:22:01][INFO] SeqIndex::SeqIndex: Done!
[2023-03-07 17:22:01][INFO] AllMappings::load_paf: Loading PAF mappings from ${HOME}/goldrush/tests/goldrush_test_golden_path.fa.test_reads.fq.paf...
[2023-03-07 17:22:01][INFO] AllMappings::load_paf: Done!
[2023-03-07 17:22:01][INFO] serve: Accepting batch names at batch_name_input
[2023-03-07 17:22:03][INFO] goldrush-edit-targeted-bfs is ready!
[2023-03-07 17:22:03][INFO] Polishing batches...
make: *** [${HOME}/anaconda3/envs/test/bin/goldrush.make:226: goldrush_test_golden_path.goldrush-edit-polished.fa] Segmentation fault (core dumped)

add tests

Debian Installation issues - memory free issue on test dataset

To get this working on my Debian system, I needed to directly insert calc_phred_avg from the btllib code into the goldrush-edit repository [I notice that you have code in goldrush with a similar function name], because my Debian system had an earlier version of btllib. Unfortunately, once I got that working and compiling, I had a memory free error:

gringer@musculus:~/install/goldrush-git/tests$ ./goldrush_test_demo.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 97.4M  100 97.4M    0     0  1705k      0  0:00:58  0:00:58 --:--:-- 1745k
Launching GoldRush
goldrush-path  -k 22 -w 16 -t 1000 -u 5 -a 1 -o 0.1 -p goldrush_test_silver_path -i test_reads.fq  -h 3 -j 4 -x10 -P 15 -d 5 -s 1011011110110111101101 -g 1e6 -b 10 -r 0.9 --silver_path -M 5 -m 20000
Using preset spaced seed
with:   
        span: 22
        weight: 16
Calculating 5 silver path(s)
Using:  
        tile length: 1000
        block size: 10
        seed patterns: 3
        threshold: 10
        base seed pattern: 1011011110110111101101
        minimum unassigned tiles: 5
        maximum assigned tiles: 1
        expected hash space: 3000000
        minimum average phred quality score: 15
        maximum average phred delta between first and second half of read: 5
        occupancy: 0.1
        jobs: 4
allocating bit vector
m_filterSize: 28473728
finished allocating bit vector
in 0.0012
opening: test_reads.fq
inserting bit vector
finished inserting bit vector
in 2.2366
src/tcmalloc.cc:333] Attempt to free invalid pointer 0x7fff774e4c60
make: *** [/usr/local/bin/goldrush:226: goldrush_test_silver_path_5.fq] Aborted (core dumped)

My computer has 64 GB main memory, and 800 GB SSD swap, so it shouldn't be having memory issues for this.

Question about warning and output

Hello! Thank you for developing this software, it runs smoothy!

I am writing to ask you two questions. First, I get the following warning many times in the same run (61 times to be exactly), with varying gap lengths

warning: scaffold gap is longer than 100 kbp: 162189
Is this expected? Should I change any parameters to deal with this? I am running the assembly with Nanopore reads (N50 reads 70Kb), and I have a some ultra long reads above 200 kb up to 800kb.

Second, the output folder is a little bit messy. I get many many files with similar names and really long ones.
Also I get many symbolic links pointing to the same files, and symbolic links that point to other symbolic links.
This is actually not a problem, but it is a little hard to know for certain which is the resulting file for assembly.
For ex. the log tells that w16_x10_golden_path.goldrush-edit-polished.span2.dist500.tigmint.fa.k40.w250.ntLink-5rounds.fa is the final output. I moved it to another folder to tidy the files, and then I realized It was a symbolic link. So then I had to search the log to find which was the original file.
I don't know if this is expected, it is a bug or something with my installation. But I hope it is useful information for the developing.

Thanks again!
Mylena

Add final GoldRush-Edit rule

The goldrush makefile is currently missing the final GoldRush-Edit on the assembly.

use new btllib

goldrush for generating UMI consensus?

Hello,

Would it be prudent to use goldrush to generate UMI consensus sequences from PacBio, assuming we can estimate the length of the consensus sequence?

Thank you

ONT read lengths mostly under 10 kb

Hi!

I have an ONT data set for a likely human-sized repeat-rich heterozygous genome with around 40x coverage. We tried various DNA extraction methods but a beta PromethION system in 2018 kept clogging - so we ended up shearing the DNA to 20 kb and then sequenced to get around clogging. Average read length was around 12 kb and many lengths go down to the 1 kb cutoff for the read set.

I assembled the reads using NextDenovo with a minimum read length of 1 kb - so all available read lengths - and it assembled to 1300 contigs with a 2.3 mB n50 and 89% BUSCO Metazoa - but 1.5 Gb assembled, so less than expected but we don't have any actual measurements or estimates yet. Overall, reasonably good - though it took 5 weeks to finish and guidance from the developers as it was much slower than expected.

I next assembled with Goldrush:

goldrush run reads=all-ont-reads G=3e9 t=60 m=1000

and it finished in a day but assembled relatively poor: 3500 contigs with a 0.3 mB n50.

I'm wondering if like for NextDenovo, it is necessary to adjust goldrush parameters for the shorter ONT read lengths. Or in general, are there parameters you might recommend in this situation or in general for testing?

I don't have a measurement or estimate of genome size for the species yet - guessing it is around human sized based on related species.

My machines are either 45 CPU and 1 Tb RAM or 60 CPU and 500 Gb RAM.

Thank you very much :)
Eric

Filename should not contain .

When trying to run Goldrush v1.0.1 installed through conda it failed to start.
Command:
goldrush run reads=filename.with.period G=6e8 t=54
Relevant part of output:

goldrush-path  -k 22 -w 16 -t 1000 -u 5 -a 1 -o 0.1 -p w16_x10_silver_path -i   -h 3 -j 54 -x10 -P 15 -d 5 -s 1011011110110111101101 -g 6e8 -b 10 -r 0.9 --silver_path -M 5 -m 20000

opening: -h
inserting bit vector
cat: invalid option -- 'h'
Try 'cat --help' for more information.
[2022-10-31 11:49:15][WARNING] -h is empty.
Gold Path requires fastq format
[2022-10-31 11:49:15][ERROR] A helper process has finished unsuccessfully:
PID: 33206
Outcome: exited with status 1

Clearly -i doesn't get populated from the reads option.
Changing the run command to
goldrush run reads=filename-without-period G=6e8 t=54
let the pipeline starts as intended.
It could be a good idea to at least mention this in the documentation.

Repeated log messages in GR-Path

For a human run (silver path stage), see log messages repeated with same number of reads processed:

finished inserting bit vector
in 3708.8420
assigning tiles
processed 30000 reads
processed 30000 reads
processed 60000 reads
processed 60000 reads

Dependencies not found with conda environment.

May I recommend a singularity container instead?

There is a precompiled binary in github, but goldrush-path is not part of them.

I've followed the following commands:

conda create --name goldrush
conda activate ---stack goldrush
conda install -c conda-forge -c bioconda --file requirements.txt

Meson fails unless you deactivate, and then reactivate the environment, but then compilation fails.

[1/26] Compiling C++ object subprojects/goldrush-edit/subprojects/sealer/sealer-gr.p/sealer.cpp.o
FAILED: subprojects/goldrush-edit/subprojects/sealer/sealer-gr.p/sealer.cpp.o
c++ -Isubprojects/goldrush-edit/subprojects/sealer/sealer-gr.p -Isubprojects/goldrush-edit/subprojects/sealer -I../subprojects/goldrush-edit/subprojects/sealer -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -std=c++17 -O0 -g -fopenmp -pthread
 -MD -MQ subprojects/goldrush-edit/subprojects/sealer/sealer-gr.p/sealer.cpp.o -MF subprojects/goldrush-edit/subprojects/sealer/sealer-gr.p/sealer.cpp.o.d -o subprojects/goldrush-edit/subprojects/sealer/sealer-gr.p/sealer.cpp.o -c ../subprojects
/goldrush-edit/subprojects/sealer/sealer.cpp
../subprojects/goldrush-edit/subprojects/sealer/sealer.cpp:6:10: fatal error: btllib/bloom_filter.hpp: No such file or directory
    6 | #include "btllib/bloom_filter.hpp"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[2/26] Compiling C++ object subprojects/goldrush-edit/subprojects/sealer/Assembly/libassembly.a.p/AssemblyAlgorithms.cc.o
FAILED: subprojects/goldrush-edit/subprojects/sealer/Assembly/libassembly.a.p/AssemblyAlgorithms.cc.o
c++ -Isubprojects/goldrush-edit/subprojects/sealer/Assembly/libassembly.a.p -Isubprojects/goldrush-edit/subprojects/sealer/Assembly -I../subprojects/goldrush-edit/subprojects/sealer/Assembly -Isubprojects/goldrush-edit/subprojects/sealer -I../su
bprojects/goldrush-edit/subprojects/sealer -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -std=c++17 -O0 -g -fPIC -fopenmp -pthread -MD -MQ subprojects/goldrush-edit/subprojects/sealer/Assembly/libassembly.a.p/AssemblyAlgorithms.cc.o -MF subp
rojects/goldrush-edit/subprojects/sealer/Assembly/libassembly.a.p/AssemblyAlgorithms.cc.o.d -o subprojects/goldrush-edit/subprojects/sealer/Assembly/libassembly.a.p/AssemblyAlgorithms.cc.o -c ../subprojects/goldrush-edit/subprojects/sealer/Assem
bly/AssemblyAlgorithms.cc
In file included from ../subprojects/goldrush-edit/subprojects/sealer/Assembly/AssemblyAlgorithms.cc:2:
../subprojects/goldrush-edit/subprojects/sealer/Common/InsOrderedMap.h:11:10: fatal error: boost/multi_index_container.hpp: No such file or directory
   11 | #include <boost/multi_index_container.hpp>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[3/26] Compiling C++ object subprojects/goldrush-edit/subprojects/sealer/Common/libcommon.a.p/ContigID.cpp.o
FAILED: subprojects/goldrush-edit/subprojects/sealer/Common/libcommon.a.p/ContigID.cpp.o
c++ -Isubprojects/goldrush-edit/subprojects/sealer/Common/libcommon.a.p -Isubprojects/goldrush-edit/subprojects/sealer/Common -I../subprojects/goldrush-edit/subprojects/sealer/Common -Isubprojects/goldrush-edit/subprojects/sealer -I../subproject
s/goldrush-edit/subprojects/sealer -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -std=c++17 -O0 -g -fPIC -fopenmp -pthread -MD -MQ subprojects/goldrush-edit/subprojects/sealer/Common/libcommon.a.p/ContigID.cpp.o -MF subprojects/goldrush-edit
/subprojects/sealer/Common/libcommon.a.p/ContigID.cpp.o.d -o subprojects/goldrush-edit/subprojects/sealer/Common/libcommon.a.p/ContigID.cpp.o -c ../subprojects/goldrush-edit/subprojects/sealer/Common/ContigID.cpp
In file included from ../subprojects/goldrush-edit/subprojects/sealer/Common/ConstString.h:4,
                 from ../subprojects/goldrush-edit/subprojects/sealer/Common/ContigID.h:4,
                 from ../subprojects/goldrush-edit/subprojects/sealer/Common/ContigID.cpp:1:
../subprojects/goldrush-edit/subprojects/sealer/Common/Hash.h:21:11: fatal error: boost/functional/hash.hpp: No such file or directory
   21 | # include <boost/functional/hash.hpp>
      |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[4/26] Compiling C++ object subprojects/goldrush-edit/subprojects/sealer/Common/libcommon.a.p/Kmer.cpp.o
FAILED: subprojects/goldrush-edit/subprojects/sealer/Common/libcommon.a.p/Kmer.cpp.o
c++ -Isubprojects/goldrush-edit/subprojects/sealer/Common/libcommon.a.p -Isubprojects/goldrush-edit/subprojects/sealer/Common -I../subprojects/goldrush-edit/subprojects/sealer/Common -Isubprojects/goldrush-edit/subprojects/sealer -I../subproject
s/goldrush-edit/subprojects/sealer -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -std=c++17 -O0 -g -fPIC -fopenmp -pthread -MD -MQ subprojects/goldrush-edit/subprojects/sealer/Common/libcommon.a.p/Kmer.cpp.o -MF subprojects/goldrush-edit/sub
projects/sealer/Common/libcommon.a.p/Kmer.cpp.o.d -o subprojects/goldrush-edit/subprojects/sealer/Common/libcommon.a.p/Kmer.cpp.o -c ../subprojects/goldrush-edit/subprojects/sealer/Common/Kmer.cpp
In file included from ../subprojects/goldrush-edit/subprojects/sealer/Common/Kmer.h:7,
                 from ../subprojects/goldrush-edit/subprojects/sealer/Common/Kmer.cpp:2:
../subprojects/goldrush-edit/subprojects/sealer/Common/Hash.h:21:11: fatal error: boost/functional/hash.hpp: No such file or directory
   21 | # include <boost/functional/hash.hpp>
      |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[5/26] Compiling C++ object subprojects/goldrush-edit/subprojects/ntedit/ntedit-gr.p/ntedit.cpp.o
FAILED: subprojects/goldrush-edit/subprojects/ntedit/ntedit-gr.p/ntedit.cpp.o
c++ -Isubprojects/goldrush-edit/subprojects/ntedit/ntedit-gr.p -Isubprojects/goldrush-edit/subprojects/ntedit -I../subprojects/goldrush-edit/subprojects/ntedit -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-d
tor -Wextra -Wpedantic -Werror -std=c++17 -O0 -g -fopenmp -pthread -MD -MQ subprojects/goldrush-edit/subprojects/ntedit/ntedit-gr.p/ntedit.cpp.o -MF subprojects/goldrush-edit/subprojects/ntedit/ntedit-gr.p/ntedit.cpp.o.d -o subprojects/goldrush-
edit/subprojects/ntedit/ntedit-gr.p/ntedit.cpp.o -c ../subprojects/goldrush-edit/subprojects/ntedit/ntedit.cpp
../subprojects/goldrush-edit/subprojects/ntedit/ntedit.cpp:13:10: fatal error: zlib.h: No such file or directory
   13 | #include <zlib.h>
      |          ^~~~~~~~
compilation terminated.
[6/26] Compiling C++ object subprojects/goldrush-edit/src/goldrush-edit-hold.p/utils.cpp.o
FAILED: subprojects/goldrush-edit/src/goldrush-edit-hold.p/utils.cpp.o
c++ -Isubprojects/goldrush-edit/src/goldrush-edit-hold.p -Isubprojects/goldrush-edit/src -I../subprojects/goldrush-edit/src -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -Wextra -Wpedantic -Werror -std=
c++17 -O0 -g -fopenmp -pthread -MD -MQ subprojects/goldrush-edit/src/goldrush-edit-hold.p/utils.cpp.o -MF subprojects/goldrush-edit/src/goldrush-edit-hold.p/utils.cpp.o.d -o subprojects/goldrush-edit/src/goldrush-edit-hold.p/utils.cpp.o -c ../su
bprojects/goldrush-edit/src/utils.cpp
In file included from ../subprojects/goldrush-edit/src/utils.cpp:1:
../subprojects/goldrush-edit/src/utils.hpp:6:10: fatal error: btllib/bloom_filter.hpp: No such file or directory
    6 | #include "btllib/bloom_filter.hpp"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[7/26] Compiling C++ object subprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p/utils.cpp.o
FAILED: subprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p/utils.cpp.o
c++ -Isubprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p -Isubprojects/goldrush-edit/src -I../subprojects/goldrush-edit/src -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -Wextra -Wpedantic -Werr
or -std=c++17 -O0 -g -fopenmp -pthread -MD -MQ subprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p/utils.cpp.o -MF subprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p/utils.cpp.o.d -o subprojects/goldrush-edit/src/goldrush-edit-tar
geted-bfs.p/utils.cpp.o -c ../subprojects/goldrush-edit/src/utils.cpp
In file included from ../subprojects/goldrush-edit/src/utils.cpp:1:
../subprojects/goldrush-edit/src/utils.hpp:6:10: fatal error: btllib/bloom_filter.hpp: No such file or directory
    6 | #include "btllib/bloom_filter.hpp"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[8/26] Compiling C++ object subprojects/goldrush-edit/src/goldrush-edit-hold.p/mappings.cpp.o
FAILED: subprojects/goldrush-edit/src/goldrush-edit-hold.p/mappings.cpp.o
c++ -Isubprojects/goldrush-edit/src/goldrush-edit-hold.p -Isubprojects/goldrush-edit/src -I../subprojects/goldrush-edit/src -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -Wextra -Wpedantic -Werror -std=
c++17 -O0 -g -fopenmp -pthread -MD -MQ subprojects/goldrush-edit/src/goldrush-edit-hold.p/mappings.cpp.o -MF subprojects/goldrush-edit/src/goldrush-edit-hold.p/mappings.cpp.o.d -o subprojects/goldrush-edit/src/goldrush-edit-hold.p/mappings.cpp.o
 -c ../subprojects/goldrush-edit/src/mappings.cpp
In file included from ../subprojects/goldrush-edit/src/seqindex.hpp:4,
                 from ../subprojects/goldrush-edit/src/mappings.hpp:4,
                 from ../subprojects/goldrush-edit/src/mappings.cpp:1:
../subprojects/goldrush-edit/src/utils.hpp:6:10: fatal error: btllib/bloom_filter.hpp: No such file or directory
    6 | #include "btllib/bloom_filter.hpp"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[9/26] Compiling C++ object subprojects/goldrush-edit/src/goldrush-edit-index.p/utils.cpp.o
FAILED: subprojects/goldrush-edit/src/goldrush-edit-index.p/utils.cpp.o
c++ -Isubprojects/goldrush-edit/src/goldrush-edit-index.p -Isubprojects/goldrush-edit/src -I../subprojects/goldrush-edit/src -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -Wextra -Wpedantic -Werror -std
=c++17 -O0 -g -fopenmp -pthread -MD -MQ subprojects/goldrush-edit/src/goldrush-edit-index.p/utils.cpp.o -MF subprojects/goldrush-edit/src/goldrush-edit-index.p/utils.cpp.o.d -o subprojects/goldrush-edit/src/goldrush-edit-index.p/utils.cpp.o -c .
./subprojects/goldrush-edit/src/utils.cpp
In file included from ../subprojects/goldrush-edit/src/utils.cpp:1:
../subprojects/goldrush-edit/src/utils.hpp:6:10: fatal error: btllib/bloom_filter.hpp: No such file or directory
    6 | #include "btllib/bloom_filter.hpp"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[10/26] Compiling C++ object subprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p/seqindex.cpp.o
FAILED: subprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p/seqindex.cpp.o
c++ -Isubprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p -Isubprojects/goldrush-edit/src -I../subprojects/goldrush-edit/src -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -Wextra -Wpedantic -Werr
or -std=c++17 -O0 -g -fopenmp -pthread -MD -MQ subprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p/seqindex.cpp.o -MF subprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p/seqindex.cpp.o.d -o subprojects/goldrush-edit/src/goldrush-ed
it-targeted-bfs.p/seqindex.cpp.o -c ../subprojects/goldrush-edit/src/seqindex.cpp
In file included from ../subprojects/goldrush-edit/src/seqindex.hpp:4,
                 from ../subprojects/goldrush-edit/src/seqindex.cpp:1:
../subprojects/goldrush-edit/src/utils.hpp:6:10: fatal error: btllib/bloom_filter.hpp: No such file or directory
    6 | #include "btllib/bloom_filter.hpp"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[11/26] Compiling C++ object subprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p/mappings.cpp.o
FAILED: subprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p/mappings.cpp.o
c++ -Isubprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p -Isubprojects/goldrush-edit/src -I../subprojects/goldrush-edit/src -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -Wextra -Wpedantic -Werr
or -std=c++17 -O0 -g -fopenmp -pthread -MD -MQ subprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p/mappings.cpp.o -MF subprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p/mappings.cpp.o.d -o subprojects/goldrush-edit/src/goldrush-ed
it-targeted-bfs.p/mappings.cpp.o -c ../subprojects/goldrush-edit/src/mappings.cpp
In file included from ../subprojects/goldrush-edit/src/seqindex.hpp:4,
                 from ../subprojects/goldrush-edit/src/mappings.hpp:4,
                 from ../subprojects/goldrush-edit/src/mappings.cpp:1:
../subprojects/goldrush-edit/src/utils.hpp:6:10: fatal error: btllib/bloom_filter.hpp: No such file or directory
    6 | #include "btllib/bloom_filter.hpp"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[12/26] Compiling C++ object subprojects/goldrush-edit/src/goldrush-edit-index.p/seqindex.cpp.o
FAILED: subprojects/goldrush-edit/src/goldrush-edit-index.p/seqindex.cpp.o
c++ -Isubprojects/goldrush-edit/src/goldrush-edit-index.p -Isubprojects/goldrush-edit/src -I../subprojects/goldrush-edit/src -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -Wextra -Wpedantic -Werror -std
=c++17 -O0 -g -fopenmp -pthread -MD -MQ subprojects/goldrush-edit/src/goldrush-edit-index.p/seqindex.cpp.o -MF subprojects/goldrush-edit/src/goldrush-edit-index.p/seqindex.cpp.o.d -o subprojects/goldrush-edit/src/goldrush-edit-index.p/seqindex.c
pp.o -c ../subprojects/goldrush-edit/src/seqindex.cpp
In file included from ../subprojects/goldrush-edit/src/seqindex.hpp:4,
                 from ../subprojects/goldrush-edit/src/seqindex.cpp:1:
../subprojects/goldrush-edit/src/utils.hpp:6:10: fatal error: btllib/bloom_filter.hpp: No such file or directory
    6 | #include "btllib/bloom_filter.hpp"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[13/26] Compiling C++ object subprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p/goldrush_edit_targeted_bfs.cpp.o
FAILED: subprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p/goldrush_edit_targeted_bfs.cpp.o
c++ -Isubprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p -Isubprojects/goldrush-edit/src -I../subprojects/goldrush-edit/src -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -Wextra -Wpedantic -Werr
or -std=c++17 -O0 -g -fopenmp -pthread -MD -MQ subprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p/goldrush_edit_targeted_bfs.cpp.o -MF subprojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p/goldrush_edit_targeted_bfs.cpp.o.d -o subpr
ojects/goldrush-edit/src/goldrush-edit-targeted-bfs.p/goldrush_edit_targeted_bfs.cpp.o -c ../subprojects/goldrush-edit/src/goldrush_edit_targeted_bfs.cpp
In file included from ../subprojects/goldrush-edit/src/seqindex.hpp:4,
                 from ../subprojects/goldrush-edit/src/mappings.hpp:4,
                 from ../subprojects/goldrush-edit/src/goldrush_edit_targeted_bfs.cpp:1:
../subprojects/goldrush-edit/src/utils.hpp:6:10: fatal error: btllib/bloom_filter.hpp: No such file or directory
    6 | #include "btllib/bloom_filter.hpp"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[14/26] Compiling C++ object subprojects/goldrush-edit/src/goldrush-edit-hold.p/seqindex.cpp.o
FAILED: subprojects/goldrush-edit/src/goldrush-edit-hold.p/seqindex.cpp.o
c++ -Isubprojects/goldrush-edit/src/goldrush-edit-hold.p -Isubprojects/goldrush-edit/src -I../subprojects/goldrush-edit/src -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -Wextra -Wpedantic -Werror -std=
c++17 -O0 -g -fopenmp -pthread -MD -MQ subprojects/goldrush-edit/src/goldrush-edit-hold.p/seqindex.cpp.o -MF subprojects/goldrush-edit/src/goldrush-edit-hold.p/seqindex.cpp.o.d -o subprojects/goldrush-edit/src/goldrush-edit-hold.p/seqindex.cpp.o
 -c ../subprojects/goldrush-edit/src/seqindex.cpp
In file included from ../subprojects/goldrush-edit/src/seqindex.hpp:4,
                 from ../subprojects/goldrush-edit/src/seqindex.cpp:1:
../subprojects/goldrush-edit/src/utils.hpp:6:10: fatal error: btllib/bloom_filter.hpp: No such file or directory
    6 | #include "btllib/bloom_filter.hpp"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[15/26] Compiling C++ object subprojects/goldrush-edit/src/goldrush-edit-index.p/goldrush_edit_index.cpp.o
FAILED: subprojects/goldrush-edit/src/goldrush-edit-index.p/goldrush_edit_index.cpp.o
c++ -Isubprojects/goldrush-edit/src/goldrush-edit-index.p -Isubprojects/goldrush-edit/src -I../subprojects/goldrush-edit/src -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -Wextra -Wpedantic -Werror -std
=c++17 -O0 -g -fopenmp -pthread -MD -MQ subprojects/goldrush-edit/src/goldrush-edit-index.p/goldrush_edit_index.cpp.o -MF subprojects/goldrush-edit/src/goldrush-edit-index.p/goldrush_edit_index.cpp.o.d -o subprojects/goldrush-edit/src/goldrush-e
dit-index.p/goldrush_edit_index.cpp.o -c ../subprojects/goldrush-edit/src/goldrush_edit_index.cpp
In file included from ../subprojects/goldrush-edit/src/seqindex.hpp:4,
                 from ../subprojects/goldrush-edit/src/goldrush_edit_index.cpp:1:
../subprojects/goldrush-edit/src/utils.hpp:6:10: fatal error: btllib/bloom_filter.hpp: No such file or directory
    6 | #include "btllib/bloom_filter.hpp"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[16/26] Compiling C++ object subprojects/goldrush-edit/src/goldrush-edit-index.p/mappings.cpp.o
FAILED: subprojects/goldrush-edit/src/goldrush-edit-index.p/mappings.cpp.o
c++ -Isubprojects/goldrush-edit/src/goldrush-edit-index.p -Isubprojects/goldrush-edit/src -I../subprojects/goldrush-edit/src -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -Wextra -Wpedantic -Werror -std
=c++17 -O0 -g -fopenmp -pthread -MD -MQ subprojects/goldrush-edit/src/goldrush-edit-index.p/mappings.cpp.o -MF subprojects/goldrush-edit/src/goldrush-edit-index.p/mappings.cpp.o.d -o subprojects/goldrush-edit/src/goldrush-edit-index.p/mappings.c
pp.o -c ../subprojects/goldrush-edit/src/mappings.cpp
In file included from ../subprojects/goldrush-edit/src/seqindex.hpp:4,
                 from ../subprojects/goldrush-edit/src/mappings.hpp:4,
                 from ../subprojects/goldrush-edit/src/mappings.cpp:1:
../subprojects/goldrush-edit/src/utils.hpp:6:10: fatal error: btllib/bloom_filter.hpp: No such file or directory
    6 | #include "btllib/bloom_filter.hpp"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[17/26] Compiling C++ object goldrush_path/goldrush-path.p/goldrush_path.cpp.o
FAILED: goldrush_path/goldrush-path.p/goldrush_path.cpp.o
c++ -Igoldrush_path/goldrush-path.p -Igoldrush_path -I../goldrush_path -I../goldrush_path/Common -I../external/btllib/include -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -Wextra -Wpedantic -Werror -st
d=c++17 -O0 -g -fopenmp -pthread -MD -MQ goldrush_path/goldrush-path.p/goldrush_path.cpp.o -MF goldrush_path/goldrush-path.p/goldrush_path.cpp.o.d -o goldrush_path/goldrush-path.p/goldrush_path.cpp.o -c ../goldrush_path/goldrush_path.cpp
In file included from ../goldrush_path/MIBFConstructSupport.hpp:21,
                 from ../goldrush_path/goldrush_path.cpp:6:
../goldrush_path/MIBloomFilter.hpp:24:10: fatal error: sdsl/bit_vector_il.hpp: No such file or directory
   24 | #include <sdsl/bit_vector_il.hpp>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.

Ie. target in goldrush makefile to skip the polishing step, in the case the user might want that flexibility

Update LICENSE

We generally use the GPL-3 license - currently, the repo has an MIT license