GPU Guidance about hypre HOT 4 CLOSED

Hi, I'm looking to use Hypre's GPU capabilities, but since this seems experimental / undocumented for now, I'm asking here if you might be able to give me some guidance. I'm working on a Fortran codebase and access Hypre's C interface through ISO C bindings. We use two interfaces: Struct's hybrid solve, and BoomerAMG with SStruct.

1. First questions are about which interfaces support which GPU backends. Is OpenMP device offloading supported for both Struct hybrid and SStruct BoomerAMG? If not, how about the CUDA backend? Worst case (for us), does the IJ interface support OpenMP device offloading or CUDA?

Hypre supports openmp offloading in the struct solvers. BoomerAMG also has some computations (very limited, basically in matvec) that have ported with openmp. The solve phase of BoomerAMG has full GPU capability with CUDA.

2. How portable is OpenMP-enabled Hypre (e.g., Nvidia, AMD, and upcoming Intel GPUs)? I noticed during build that --with-cuda seems to be required when --with-device-openmp is enabled. For now I only need to support Nvidia GPUs, but would like to keep portability in mind.

--with-cuda is not required with --with-device-openmp and actually they should not be used together.
It should be --with-device-openmp --enable-unified-memory. OpenMP 4.5 is one of our solutions to resolve the portability issue but we have not tested hypre's GPU capability on other types of GPUs than the Nvidia ones.

3. Should memory already be on the GPU when we call Hypre routines? If so, does this only apply to matrix and vector values, or to the grid, graph, and stencil as well?

Yes, the memory should be on the GPU and it only applies to matrices and vectors. The memory allocation should be handled internally by the hypre user-level functions such as HYPRE_StructMatrixInitialize, HYPRE_StructVectorInitialize, HYPRE_IJVectorInitialize, HYPRE_IJMatrixInitialize, etc

4. Do you have any examples or documentation you could point me to for GPU-enabled Hypre use?

We are working on the GPU examples and documentation. Currently, you can look at test/ij.c and test/struct.c for GPU-enabled Hypre use

Thanks!

from hypre.

Thank you, I was able to get our struct-interface systems running on the GPU using CUDA. I'm going to try using OpenMP too now that I can see what to do.

I haven't been able to get the semi-struct to work though. Does the BoomerAMG solver work with CUDA semi-struct objects? I am getting the error:

Segfault
CUDA Error Code[101]: invalid device ordinal
 hypre_memory.c(hypre_UnifiedMalloc) Line:210

I'm linking with AMReX which gives me this backtrace:

 0: htpc/test/test_htpc_solver_type() [0x55a626]
    amrex::BLBackTrace::print_backtrace_info(_IO_FILE*) at /opt/src/amrex/amrex/Src/Base/AMReX_BLBackTrace.cpp:108

 1: htpc/test/test_htpc_solver_type() [0x55b5a8]
    amrex::BLBackTrace::handler(int) at /opt/src/amrex/amrex/Src/Base/AMReX_BLBackTrace.cpp:55

 2: /lib64/libc.so.6(+0x38600) [0x1517d8da4600]
    ?? ??:0

 3: /home/zjibben/opt/hypre-cuda/lib/libHYPRE-2.17.0.so(CheckError+0x3e) [0x1517d9b1c3fe]
    ?? ??:0

 4: /home/zjibben/opt/hypre-cuda/lib/libHYPRE-2.17.0.so(hypre_CAlloc+0x22f) [0x1517d9b1f3ef]
    ?? ??:0

 5: /home/zjibben/opt/hypre-cuda/lib/libHYPRE-2.17.0.so(hypre_CSRMatrixInitialize+0x96) [0x1517d99cbe26]
    ?? ??:0

 6: /home/zjibben/opt/hypre-cuda/lib/libHYPRE-2.17.0.so(hypre_ParCSRMatrixInitialize+0x12) [0x1517d99aa632]
    ?? ??:0

 7: /home/zjibben/opt/hypre-cuda/lib/libHYPRE-2.17.0.so(hypre_IJMatrixInitializeParCSR+0x45) [0x1517d9899355]
    ?? ??:0

 8: /home/zjibben/opt/hypre-cuda/lib/libHYPRE-2.17.0.so(HYPRE_IJMatrixInitialize+0x45) [0x1517d98969e5]
    ?? ??:0

 9: /home/zjibben/opt/hypre-cuda/lib/libHYPRE-2.17.0.so(hypre_SStructUMatrixInitialize+0x672) [0x1517d9a34872]
    ?? ??:0

10: /home/zjibben/opt/hypre-cuda/lib/libHYPRE-2.17.0.so(HYPRE_SStructMatrixInitialize+0x2e2) [0x1517d9a2cf52]
    ?? ??:0

from hypre.

Has this issue been resolved? Thanks! -Rob

from hypre.

I think it's fine to close this. If I encounter something unexpected on a more recent version of Hypre, I can open a new, more focused issue. Thanks!

from hypre.