RuntimeError: Found no NVIDIA driver on your system. about openllm HOT 5 CLOSED

did you mount the GPU to the container?

from openllm.

Ok, when I'm runned it from powershell with --gpus all property (instead of GUI), this error was gone, but heres another:

ValueError: No available memory for the cache blocks. Try increasing gpu_memory_utilization when initializing the engine.

Is it possible that this model will not work with this VGA?

from openllm.

You can set --gpu-memory-utlization to 0.5

from openllm.

I'm trying, with this:

ENTRYPOINT openllm start HuggingFaceH4/zephyr-7b-alpha --backend vllm --gpu-memory-utilization 0.5

But it cause the following error:

openllm_core.exceptions.OpenLLMException: Failed to initialise vLLMEngine due to the following error:
CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.

Then:

File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 109, in init
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 112.00 MiB. GPU 0 has a total capacty of 8.00 GiB of which 0 bytes is free. Process 481 has 17179869184.00 GiB memory in use. Of the allocated memory 12.74 GiB is allocated by PyTorch, and 13.52 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

from openllm.

I've connected another RTX 3070 to my PC, and the memory issue has been resolved. However, a new problem has emerged, for which I am opening a new issue. I am closing this one since the original problem has been solved.

from openllm.