foundation-model-stack / fms-fsdp Goto Github PK
View Code? Open in Web Editor NEW๐ Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash attention v2.
Home Page: https://pytorch.org/docs/stable/fsdp.html
License: Apache License 2.0