Caught amgx exception: Cannot allocate pinned memory about amgx HOT 3 OPEN

@AnjaliSandip
It seems error indicates that pinned memory pool cannot be allocated:

Caught amgx exception: Cannot allocate pinned memory
 at: /home/anjali.sandip/ISSM/ISSM/externalpackages/petsc/src/arch-linux-c-opt/externalpackages/git.amgx/src/global_thread_handle.cu:374

It's size is currently fixed to 100 MB: https://github.com/NVIDIA/AMGX/blob/v2.4.0/src/global_thread_handle.cu#L51 regardless of the input data ( and this allocation happens during resources creation at which point we don't know problem size)

Is your process allowed to allocate page-locked memory? (i.e. for docker containers you have to provide respective ulimit flag, i.e.: --ulimit memlock=-1)

from amgx.

from amgx.

@AnjaliSandip sorry for the delayed reply.
I'm not familiar with PETSc internals, but unless PETSc environment somehow hooks cudaMallocHost, it's settings shouldn't affect AMGX, since AMGX using a call directly to CUDA Runtime: https://github.com/NVIDIA/AMGX/blob/v2.4.0/src/global_thread_handle.cu#L378

You can try running an example that tries to allocate same amount of pinned memory to see if it's environment issue, something like this: https://godbolt.org/z/7ab86qc34

If there is no obvious/easy fix to page locked memory, I would suggest opening a ticket for PETSc (https://gitlab.com/petsc/petsc/-/issues), as they are more knowledgeable about PETSc details that might be important here. You can link this issue for the reference and i can follow up in the case there would be any further questions to AMGX.

from amgx.