torch.cuda.OutOfMemoryError: CUDA out of memory about pyllama HOT 9 CLOSED

Hello, i am using a Nvidia Gtx1650 4GB GPU. Are there any way to run the 7B model on it?

Sure. I am still working on it... so that we can run 13B on a 4GB GPU.

from pyllama.

I changed the max_seq_len from 1024 to 512, and successfully inference the result in the RAM 16GB.

from pyllama.

• I simply sorted out the code of the repo, and only kept the two simplest use cases.
• Hope it can help everyone.
• LLaMADemo

from pyllama.

For me, the inference.py use 15.7GB VRAM with everything by default, though I am using Windows WSL2 which means the VRAM usage of the script should be roughly 14GB... Seems still not enough for 12GB VRAM.

from pyllama.

Oh ok, yeah that would make sense. Do you think there might be a way to run the script in a way the uses less VRAM?

from pyllama.

You can try the 8bit (less precision) model: https://github.com/tloen/llama-int8

from pyllama.

I found https://github.com/qwopqwop200/GPTQ-for-LLaMa which turn it to 4-bit, I can run the benchmark in that repo. But the code in that repo is using huggingface transformer and it seems that at least the model loading is different.

from pyllama.

I changed the max_seq_len from 1024 to 512, and successfully inference the result in the RAM 16GB.

pyllama/inference.py

Line 72 in 450d686

max_seq_len=1024,

Thanks for the tips!

from pyllama.

Hello, i am using a Nvidia Gtx1650 4GB GPU. Are there any way to run the 7B model on it?

from pyllama.