is it possible to support Complex Double data type ? about marlin HOT 5 OPEN

hi @gocanal,
I believe that for Matrix Multiplication Complex Double could be well supported without too much change in the algorithm level, since Breeze itself supports Complex Double data type. But I am not quite sure for Inversion, given that currently the inversion is based on LU Decomposition of the original matrix. If you do need to deal with Complex Matrix Inverse and your use case is not so performance critical, I think you can separate the original Complex Matrix into two parts, Real and Imaginary, each alone using a Double Matrix. Then you can deal with Complex Matrix Inverse at a higher level without a single change in the current codebase.

Best regards.
xingkungao

from marlin.

Hi,
I assume you mean we can do a 2M * 2M double data inversion, where the original matrix is M * M double complex, and the 2M * 2M matrix is:

real              imaginary
-imaginary            real

Am I correct ?

thanks
canal

from marlin.

Hi,
Yes, I mean for a M * M complex matrix (A + iB), consider it's inverse let's say (C + iD), we have
(A+iB)(C+iD) = I, and so

  AC - BD = I    
  AD + BC = 0

So we first get the inverse of 2M * 2M Double matrix

    A     -B            
    B      A     

then multiplies the inverse with the 2M * M matrix:

    I        
    0    

and we get

    C
    D   

which is exactly what we want.

hope it helps!
xingkungao

from marlin.

Thank you very much for confirming.

I did a quick test, and compared with Scalapack based solution. I am running on one machine, with only 4GB RAM (I3, 3.3GHz, Ubuntu 14.04, 64-bit Java).

For multiplication, Marlin can support up to 5k * 5k. Scalapack can support up to 8k.

For inverse, Marlin can only support up to 2.5k matrix, while Scalapack can still support up to 8k. Beyond 2.5K, marlin hits 'OutOfMemoryError: Java Heap' (the error happened in org.spark_project.protobuf.AbstractMe
ssageLite.toByteArray).

It seems that Marlin (or I should say Hadoop/Spark) consumes a lot more memory ? Or if there is something I did not configure correctly (I set spark.akka.frameSize=1024, executor-memory 2700m).

from marlin.

Sorry for late reply.

Actually, by default, 60% memory of Spark executor process is occupied by Spark BlockManager, and 20% memory of Spark executor process is used during shuffle, these two parts are configured by parameter spark.storage.memoryFraction and spark.shuffle.memoryFraction. Besides, JVM-based program consumes more memory than MPI program in general. I think 2.7GB executor memory is a bit small.

from marlin.