Comments (9)
Hello, i am using a Nvidia Gtx1650 4GB GPU. Are there any way to run the 7B model on it?
Sure. I am still working on it... so that we can run 13B on a 4GB GPU.
from pyllama.
I changed the max_seq_len
from 1024 to 512, and successfully inference the result in the RAM 16GB.
Line 72 in 450d686
from pyllama.
- I simply sorted out the code of the repo, and only kept the two simplest use cases.
- Hope it can help everyone.
- LLaMADemo
from pyllama.
For me, the inference.py
use 15.7GB VRAM with everything by default, though I am using Windows WSL2 which means the VRAM usage of the script should be roughly 14GB... Seems still not enough for 12GB VRAM.
from pyllama.
Oh ok, yeah that would make sense. Do you think there might be a way to run the script in a way the uses less VRAM?
from pyllama.
You can try the 8bit (less precision) model: https://github.com/tloen/llama-int8
from pyllama.
I found https://github.com/qwopqwop200/GPTQ-for-LLaMa which turn it to 4-bit, I can run the benchmark in that repo. But the code in that repo is using huggingface transformer and it seems that at least the model loading is different.
from pyllama.
I changed the
max_seq_len
from 1024 to 512, and successfully inference the result in the RAM 16GB.Line 72 in 450d686
Thanks for the tips!
from pyllama.
Hello, i am using a Nvidia Gtx1650 4GB GPU. Are there any way to run the 7B model on it?
from pyllama.
Related Issues (20)
- Why are params.json empty? HOT 5
- Does this include the GPTQ quantization tricks?
- Randomly get shape mismatch error
- Try Modular - Mojo
- gptq github HOT 4
- parameter inncorrect when I run make command
- an operation was attempted on something that is not a socket
- shape mismatch error
- RecursionError: maximum recursion depth exceeded while calling a Python object
- RuntimeError: Error(s) in loading state_dict for LLaMAForCausalLM: Unexpected key(s) in state_dict:
- Download watchdog kicking in? (M1 mac)
- Download 7B model seems stuck HOT 9
- evaluating has an extremely large value when quantize to 4bit. HOT 1
- NVMLError_NoPermission: Insufficient Permissions
- no module named llama HOT 1
- 12GB card HOT 2
- How to run an interactive mode in Jupyter?
- Quick Question
- torch.distributed.elastic.multiprocessing.errors.ChildFailedError HOT 1
- Question regarding EnergonAI repo
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyllama.