Flash Attention for fine-tuning about llm-workshop HOT 4 CLOSED

Hello, we have updated the code to use the FA2 support from the 🤗 transformers instead of the monkey patching. The Transformers support should handle packing as well as non packing scenarios resolving this issue.

from llm-workshop.

Great work adding it in transformers @susnato ! That PR is part of the newest release (4.35.0) you can just do pip install -U transformers

from llm-workshop.

@prince14322 and @pacman100

I am getting the error:

ValueError: The current architecture does not support Flash Attention 2.0. Please open an issue on GitHub to request support for this architecture: https://github.com/huggingface/transformers/issues/new
[2023-10-29 08:38:36,863] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0

What is going with why @pacman100 was able to run the code back in August, but now it is broken when we install the latest transformers from the git main line? What is the solution to fixing this?

from llm-workshop.

Hi @prince14322 , Flash Attention is now added to StarCoder. PR
Please install from transformers main and and you will be able to use it for starcoder.

from llm-workshop.