Possibility for MPI integration... about vg HOT 4 CLOSED

Hi Paul,

This would be interesting but isn't something I have pursued. Amdahl's law
prevents us from going much faster in every parallel step except mapping,
and mapping can be made as parallel as we want if we divide up the input
and run independent processes per chunk.

You could make things faster in terms of indexing and construction if the
reference system were broken up into tiny fragments. Again, Amdahl's law
prevents things from going faster than the serialization time of the entire
graph, so we are limited by that if we wanted to concatenate the fragments
into one stream. A similar effect would apply for the indexing. We can
shard the kmer space across a huge number of nodes. But is this necessary
for any other reason than to make construction and indexing really fast?
(Probably not.)

In summary, there are conceptual problems which are much more limiting. For
example, banded alignment of large contigs, step-wise multiple sequence
alignment, or graph/graph alignment would all help to improve the capacity
of the method. For performance improvements, I'm inclined to work towards
GCSA integration.

Hope this clarifies,

Erik
On Jun 3, 2015 3:13 AM, "Paul Grosu" [email protected] wrote:

Hi Erik,

Just out of curiosity, have you tried to integrate with MPI to work across
multiple machines. MPI plays really well with OpenMP, and would allow
access to even more cores to speed up things.

Just a thought,
Paul

—
Reply to this email directly or view it on GitHub
#38.

from vg.

@pgrosu If you're interested in trying this, I'd say go for it. I just wanted to issue a word of caution because I think there are other things to work on that may be much more fruitful.

from vg.

Hi Erik,

I apologize for not replying back yet, since I am still working on the draft for the post to look at each aspect - some posts take me several days to write up properly. Hopefully I will finish it in the next few days. I've been exposed to Amdahl's law quite a bit, though like you mentioned there might be possibilities with specific setups where limits will eventually be reached.

I always like interesting problems, but is it that you see the issues with them being dynamic programming problems where O() complexity becomes higher than preferred? Maybe there is one specific hard and important problem you could expand on, in case I might have seen a different approach that might optimize it.

Thank you,
Paul

from vg.

In the interest of keeping the issues list short, I'll pass on this. If you can find a way to run OMP programs using MPI then I think it would enable support.

from vg.