Any Option for bamsort in Parallel? about biobambam2 HOT 6 OPEN

It works quite nicely. See ablab/spades#67 (comment) for 3 shell lines and an example performance on LustreFS. Best is to have a ramdisk on the machine, write the sorted BAM file to it and then, move the final BAM with its index to the real storage filesystem.

from biobambam2.

The sortthreads option should work. I will add it to the documention in the next version.

from biobambam2.

So how does sortthreads relate to inputthreads and outputthreads if I use all three on the commandline? In which ratio should I distribute the available cores in between these three?

from biobambam2.

@mmokrejs: the issue with bamsort is that it does not use pooled threading throughout the program. The input, output and sortthreads may run all at the same time. You can use a tool like cpuset to limit the number of real cores used by the program and set all three to that number of threads. If you want a program that will use exactly a given number of threads for processing at any time, then please check bamsormadup, it was designed for this.

from biobambam2.

@gt1 You say that if I run bamsort sortthreads=$phys_cores inputthreads=$phys_cores outputthreads=$phys_cores that I will end-up with load 300?

Shall I divide the numbers of available cores by 3 to ensure the load will be max 100?

But isn't one decompression and one compression thread enough? So bamsort sortthreads=$phys_cores-2 inputthreads=1 outputthreads=1 ?

from biobambam2.

@mmokrejs This could happen, although it is rather unlikely. In my experience, assuming you do not set level=0 for uncompressed output, the output compression is rather compute heavy, so you might want to spend more threads there.

from biobambam2.