torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 432.00 MiB (GPU 2; 23.65 GiB total capacity; 20.88 GiB already allocated; 259.56 MiB free; about openchatkit HOT 6 OPEN

It'd be really cool if the minimum requirements of the model (size on disk for data set, vram requirements) on the readme, that would save a lot of people some time.

from openchatkit.

You ran out of GPU memory. Describe more on your setup like what you are using and what command you ran to resolve.

from openchatkit.

It'd be really cool if the minimum requirements of the model (size on disk for data set, vram requirements) on the readme, that would save a lot of people some time.

That's a great idea. I'll put up a PR soon to document this.

from openchatkit.

You ran out of GPU memory. Describe more on your setup like what you are using and what command you ran to resolve.

batch_size=4

from openchatkit.

(OpenChatKit) root@aca2869c8358:~/OpenChatKit-main# python inference/bot.py
Loading /root/OpenChatKit-main/inference/../huggingface_models/GPT-NeoXT-Chat-Base-20B to cuda:0...
Traceback (most recent call last):
File "/root/OpenChatKit-main/inference/bot.py", line 185, in
main()
File "/root/OpenChatKit-main/inference/bot.py", line 181, in main
).cmdloop()
File "/root/anaconda3/envs/OpenChatKit/lib/python3.10/cmd.py", line 105, in cmdloop
self.preloop()
File "/root/OpenChatKit-main/inference/bot.py", line 64, in preloop
self._model = ChatModel(self._model_name_or_path, self._gpu_id)
File "/root/OpenChatKit-main/inference/bot.py", line 24, in init
self._model.to(device)
File "/root/anaconda3/envs/OpenChatKit/lib/python3.10/site-packages/torch/nn/modules/module.py", line 989, in to
return self._apply(convert)
File "/root/anaconda3/envs/OpenChatKit/lib/python3.10/site-packages/torch/nn/modules/module.py", line 641, in _apply
module._apply(fn)
File "/root/anaconda3/envs/OpenChatKit/lib/python3.10/site-packages/torch/nn/modules/module.py", line 641, in _apply
module._apply(fn)
File "/root/anaconda3/envs/OpenChatKit/lib/python3.10/site-packages/torch/nn/modules/module.py", line 641, in _apply
module._apply(fn)
[Previous line repeated 2 more times]
File "/root/anaconda3/envs/OpenChatKit/lib/python3.10/site-packages/torch/nn/modules/module.py", line 664, in _apply
param_applied = fn(param)
File "/root/anaconda3/envs/OpenChatKit/lib/python3.10/site-packages/torch/nn/modules/module.py", line 987, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 288.00 MiB (GPU 0; 14.56 GiB total capacity; 13.86 GiB already allocated; 90.44 MiB free; 13.88 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

from openchatkit.

some problem, any idea how much memory it needs? or any solution to reduce the memory use? Thanks.

from openchatkit.