Comments (4)
I think you need at least 14GB GPU memory to load the 7b model in fp16.
from wanda.
@Eric-mingjie Thanks Eric, mine is 24 GB GPU memory. Given that at least 14GB would be used to load the model. I still have ~10 GB left in Nvidia L4. Are there any extra activities taking more memory and can we avoid in the arguments?
from wanda.
Mine has 80GB of GPU RAM >>>>>NVIDIA A100 (and H100) GPU in Stanage has 80GB of GPU RAM
still got this error.
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 512.00 MiB. GPU
complete error for reference:
torch 2.3.0
transformers 4.41.0.dev0
accelerate 0.31.0.dev0
of gpus: 1
loading llm model mistralai/Mistral-7B-Instruct-v0.2
^MLoading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]^MLoading checkpoint shards: 33%|███▎ | 1/3 [00:12<00:24, 12.09s/it]^MLoading checkpoint shards: 67%|██████▋ | 2/3 [00:29<00:15,$
use device cuda:0
pruning starts
loading calibdation data
dataset loading complete
Traceback (most recent call last):
File "/mnt/parscratch/users/acq22stk/teamproject/wanda/main.py", line 110, in
main()
File "/mnt/parscratch/users/acq22stk/teamproject/wanda/main.py", line 69, in main
prune_wanda(args, model, tokenizer, device, prune_n=prune_n, prune_m=prune_m)
File "/mnt/parscratch/users/acq22stk/teamproject/wanda/lib/prune.py", line 160, in prune_wanda
outs[j] = layer(inps[j].unsqueeze(0), attention_mask=attention_mask, position_ids=position_ids)[0]
File "/users/acq22stk/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/users/acq22stk/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/users/acq22stk/.conda/envs/prune_llm/lib/python3.9/site-packages/transformers/models/mistral/modeling_mistral.py", line 754, in forward
hidden_states = self.input_layernorm(hidden_states)
File "/users/acq22stk/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/users/acq22stk/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/users/acq22stk/.conda/envs/prune_llm/lib/python3.9/site-packages/transformers/models/mistral/modeling_mistral.py", line 85, in forward
hidden_states = hidden_states.to(torch.float32)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 512.00 MiB. GPU
from wanda.
I have the same error with the Mixtral 8x7B model using 4 A6000 GPUs (48GiB memory per device).
from wanda.
Related Issues (20)
- llama_7b wikitext perplexity 7.0915350914 HOT 3
- OPT-66B, unstructured sparsity gets wikitext perplexity 3404.0751953125 HOT 1
- Perplexity is off for Llama 2-7b HOT 4
- Where is cache['attention_mask']? HOT 1
- question about the code in the sparse_trainer.py HOT 2
- Compressing a Finetuned llama2 model with lora HOT 1
- AttributeError: 'NoneType' object has no attribute 'to' HOT 6
- Cannot reproduce Llama2 results HOT 5
- why choosing vicuna as the tokenizer HOT 2
- Ambiguous result for LLAMA-2-13b HOT 3
- which dataset do you use for jailbreak?
- How to save the pruned model?
- Llama2 `ExpectedMoreSplits` Exception HOT 1
- Breaks under Numpy 2
- Error when trying to run Llama2-7b: attention_mask and position_ids are None HOT 1
- AttributeError: module 'lm_eval.tasks' has no attribute 'ALL_TASKS' HOT 1
- Results After LoRA Fine-Tuning
- Why did I use llama2-7B when pruningthe model to the same size as the original model
- Llama3 models
- AttributeError: 'NoneType' object has no attribute 'to' when using wanda pruning HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from wanda.