Multi-GPU bug? about alpaca-lora HOT 11 OPEN

This code doesn't work on multi-GPU yet; I'm still running it on my single RTX 4090. Might adapt to multi-GPU in a bit to speed up training.

(Also please git pull if you haven't recently; I fixed a bug in the dataset generation code)

from alpaca-lora.

Just accelerate launch finetune.py.
It works.

from alpaca-lora.

The PEFT code needs to be adapted to make better use of accelerate. I think there are some examples of how to do it in the huggingface/peft repo but I can't test them as I don't have a multi-GPU setup myself.

from alpaca-lora.

Might adapt to multi-GPU in a bit to speed up training.
Could you point me to the reason why this is not working on multiple GPUs? I.e., which part is breaking it? The LoRA stuff or the 8-bit stuff? Something else? Thanks!

from alpaca-lora.

The PEFT code needs to be adapted to make better use of accelerate. I think there are some examples of how to do it in the huggingface/peft repo but I can't test them as I don't have a multi-GPU repo myself.

That would be neat. Kaggle already offers 2xT4 with 2x16GB RAM, which would be probably quite slow but probably enough to train 13B model.

from alpaca-lora.

this may work for you: https://discord.com/channels/1086739839761776660/1087706061022187641/1087944493564698674

This link seems to be a private link. What is it about?

from alpaca-lora.

I use the following command to run normally on multi-gpu
CUDA_VISIBLE_DEVICES=0,1,2,3 accelerate launch finetune.py`
But actually only gpu 0 and 1 are used.
I don't know why

from alpaca-lora.

I use only one GPU, add some codes in beginning

gpu_list = [7]
gpu_list_str = ','.join(map(str, gpu_list))
os.environ.setdefault("CUDA_VISIBLE_DEVICES", gpu_list_str)
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

device assign the one GPU to torch

from alpaca-lora.

I use the following command to run normally on multi-gpu CUDA_VISIBLE_DEVICES=0,1,2,3 accelerate launch finetune.py` But actually only gpu 0 and 1 are used. I don't know why

Try to enable all the GPUs by running accelerate config

from alpaca-lora.

fwiw solution for me was to not use torchrun to launch the script. I was having an issue where single-GPU training worked fine, but with multi-GPU training after a single update step, the model would freeze -- gpu-util was at 100% but no more updates happened. Getting rid of torchrun and simply calling the python script solved it and seems to use DDP fine.

would be great to have more guidance on what kinds of setups work for launching multi-GPU jobs. would be happy to contribute information about my setup for that as well :)

from alpaca-lora.

fwiw solution for me was to not use torchrun to launch the script. I was having an issue where single-GPU training worked fine, but with multi-GPU training after a single update step, the model would freeze -- gpu-util was at 100% but no more updates happened. Getting rid of torchrun and simply calling the python script solved it and seems to use DDP fine.

would be great to have more guidance on what kinds of setups work for launching multi-GPU jobs. would be happy to contribute information about my setup for that as well :)

Are you able to do multi-GPU inferencing?

from alpaca-lora.