Hello from HF Diffusers about flash-linear-attention HOT 3 OPEN

Hello Sayak,

Thanks for your interest! Unfortunately, we cannot directly convert a model with softmax attention to one with linear attention during inference without any additional training. However, it is indeed possible to finetune pretrained LLMs for a few steps—much fewer than training from scratch—to switch from regular attention to linear attention. You can refer to these resources for more details: arXiv:2405.06640, OpenReview, etc.

from flash-linear-attention.

@sayakpaul FYI, we release some weights converted from Mistral-7B-v0.1 as in arXiv:2405.06640. You can have a try by loading fla-hub/gla-7B-mistral-20B, fla-hub/gsa-7B-mistral-20B or fla-hub/gsa-7B-mistral-100B

from flash-linear-attention.

Would there be any interest to try something similar for diffusion models? I am happy to team up. I believe this will be a very significant contribution to the community now that large models like Flux are coming up.

from flash-linear-attention.