Cuda support is not working about node-llama-cpp HOT 3 CLOSED

@pankaj-kumar34 I was having issues with CUDA support as well, you need to enable the cuda flag when downloading/building:

npx node-llama-cpp download --cuda

It would compile successfully but I would get a warning saying something about cuBLAS not being detected, so to fix that I installed the CUDA toolkit: https://developer.nvidia.com/cuda-downloads

Then when I compiled it would say -- cuBLAS found but I would get an error:

CMake Error at ~/.local/lib/python3.10/site-packages/cmake/data/share/cmake-3.26/Modules/CMakeDetermineCUDACompiler.cmake:603 (message):
  Failed to detect a default CUDA architecture.



  Compiler output:

Call Stack (most recent call first):
  llama.cpp/CMakeLists.txt:285 (enable_language)


-- Configuring incomplete, errors occurred!

To fix this you need to set environment to the directory:

export CUDACXX=/usr/local/cuda-12.2/bin/nvcc

Note that your CUDA version may vary. Once this is done you can run the download command again and it should be successful.

Then when defining the Llama model enable gpuLayers:

const model = new LlamaModel({
	modelPath,
	gpuLayers: 64 // Or whatever makes sense.
});

When running you should be able to see an output similar to:

llm_load_tensors: ggml ctx size =    0.09 MB
llm_load_tensors: using CUDA for GPU acceleration
llm_load_tensors: mem required  =   41.11 MB (+ 2048.00 MB per state)
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloading v cache to GPU
llm_load_tensors: offloading k cache to GPU
llm_load_tensors: offloaded 35/35 layers to GPU
llm_load_tensors: VRAM used: 4741 MB

You can monitor GPU usage with:

watch -d nvidia-smi

from node-llama-cpp.

@pankaj-kumar34 There are many environment variables that you can set during the build and runtime to configure the CUDA support of llama.cpp; you can find more details here.

I do not have an Nvidia GPU, so I can't test how well (if at all) the CUDA support works or provide better documentation for this process.

You're welcome to play around with the code to try making it work and share your conclusions here or open a PR to improve the CUDA support of this node module.

from node-llama-cpp.

It's working as expected.

from node-llama-cpp.