Comments (6)
@marella it appears that was the issue:
Traceback (most recent call last):
File "<string>", line 1, in <module>
NameError: name 'CUDA_PATH' is not defined
When adding the following code to my python file, it resolved the issue:
import os
cuda_path = "C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8"
os.environ["CUDA_PATH"] = cuda_path
Interesting note -- using CPU only gives a short answer. Same question with the GPU involved is more verbose!
from ctransformers.
In the latest version 0.2.21, precompiled libraries are provided for CUDA 12, so no compilation is required and it no longer needs CUDA to be installed at system level:
pip install ctransformers ctransformers[cuda]
llm = AutoModelForCausalLM.from_pretrained("TheBloke/Llama-2-7B-GGML", gpu_layers=50)
See example in Colab.
Feel free to re-open if you are still facing an issue.
from ctransformers.
Hi, is the normal installation without CUBLAS (pip install ctransformers
) working?
What is your Python version? Can you check if CUDA_PATH
is set:
python3 -c 'import os; print(os.environ["CUDA_PATH"])'
from ctransformers.
Same issue here
from ctransformers.
Thanks! I think I checked path and env vars but I can't recall for sure! I'll try and provide more context
from ctransformers.
I get this error while running the colab code :
CUDA error 35 at D:\a\ctransformers\ctransformers\models\ggml\ggml-cuda.cu:5067: CUDA driver version is insufficient for CUDA runtime version
please help
Windows 11
CUDA 11.8
RTX 3090
nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 522.25 Driver Version: 522.25 CUDA Version: 11.8 |
|-------------------------------+----------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... WDDM | 00000000:41:00.0 On | N/A |
| 0% 41C P8 25W / 420W | 716MiB / 24576MiB | 3% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_Oct_11_22:11:21_Pacific_Daylight_Time_2021
Cuda compilation tools, release 11.4, V11.4.152
Build cuda_11.4.r11.4/compiler.30521435_0
from ctransformers.
Related Issues (20)
- precompiled rocm and metal wheels
- Multimodal models compatibility
- Pulling models outside of hf?
- Unsupported Model : Zephyr 'stablelm' GGUF
- Add support for Microsoft Phi-2
- Not working with gpu_layers HOT 2
- Cannot generate text on GPU HOT 3
- How to load the finetuned model in safetensors format(not in gguf)
- Does ctransformers boost the inference speed in llm inference?
- Add support for Google's Gemma models
- Does ctransformers support ollama models? HOT 1
- Inputting embeddings directly
- GGUF MODEL INFERENCE
- Add Support for Google/Gemma-2b-it
- Error when trying to run on kali linux
- OSError: .......cannot open shared object file: No such file or directory HOT 3
- Support for Llama3 HOT 2
- Problem accessing libctransformers.so HOT 1
- model not loading on GPU
- Link in readme broken
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ctransformers.