This monorepo collects libraries of packages that accelerate fine-tuning / training of large models, intended to be part of the fms-hf-tuning suite.
This package is in BETA under extensive development. Expect breaking changes!
Plugin | Description | Depends | License | Status |
---|---|---|---|---|
framework | This acceleration framework for integration with huggingface trainers | Beta | ||
accelerated-peft | For PEFT-training, e.g., 4bit QLoRA. | Huggingface AutoGPTQ |
Apache 2.0 MIT |
Beta |
TBA | Unsloth-inspired. Fused LoRA and triton kernels (e.g., fast cross-entropy, rms, rope) | Xformers | Apache 2.0 with exclusions. | Under Development |
TBA | MegaBlocks inspired triton Kernels and acclerations for Mixture-of-Expert models | Apache 2.0 | Under Development |
This is intended to be a collection of many acceleration routines (including accelerated peft and other techniques). Below demonstrates a concrete example to show how to accelerate your tuning experience with tuning/sft_trainer.py from fms-hf-tuning
.
Below instructions for accelerated peft fine-tuning. In particular GPTQ-LoRA tuning with the AutoGPTQ triton_v2
kernel; this kernel is state-of-the-art provided by jeromeku
on Mar 2024:
- Checkout fms-hf-tuning and install the framework library:
or alternatively install the framework directly:
pip install -e .[fms-accel]
pip install git+https://github.com/foundation-model-stack/fms-acceleration.git#subdirectory=plugins/framework
- The above installs the command line utility
fms_acceleration.cli
, which can then be used to install plugins. Uselist
to view available plugins; this list updates as more plugins get developed:and then$ python -m fms_acceleration.cli list Choose from the list of plugin shortnames, and do: * 'python -m fms_acceleration.cli install <pip-install-flags> PLUGIN_NAME'. List of PLUGIN_NAME [PLUGIN_SHORTNAME]: 1. fms_acceleration_peft [peft]
install
the plugin. We install thefms-acceleration-peft
plugin for GPTQ-LoRA tuning with triton v2 as:The above is the equivalent of:python -m fms_acceleration.cli install fms_acceleration_peft
pip install git+https://github.com/foundation-model-stack/fms-acceleration.git#subdirectory=plugins/accelerated-peft
- Prepare a YAML configuration for the acceleration framework plugins
- Full sample configuration list shows the
plugins
required for the configs. - Accelerated GPTQ-LoRA configuration here.
- Once
fms-acceleration-peft
plugin was installed (recall above), the framework uses the plugins given the framework configuration; for more details see framework/README.md.
- Full sample configuration list shows the
- Run
sft_trainer.py
with the following arguments:-
--acceleration_framework_config_file
pointing to framework configuration YAML. -
arguments required for correct operation (e.g., if using accelerated peft, then
peft_method
is required).- For sample configurations, these arguments can be referred from defaults.yaml and scenarios.yaml.
- Arguments not critical to the plugins found in defaults.yaml. These can be taken with liberty.
- Arguments critcal to plugins found in scenarios.yaml. The relevant section of scenarios.yaml, is the one whose
framework_config
entries, match theshortname
of the sample configuration of interest.
-
More info on
defaults.yaml
andscenarios.yaml
found here.# when using sample-configurations, arguments can be referred from # defaults.yaml and scenarios.yaml python sft_trainer.py \ --acceleration_framework_config_file sample-configurations/acceleration_framework.yaml \ ... # arguments from default.yaml ... # arguments from scenarios.yaml
-
Over time, more plugins will be updated, so please check here for the latest accelerations!.
This repo requires CUDA to compute the kernels, and it is convinient to use NVidia Pytorch Containers that already comets with CUDA installed. We have tested with the following versions:
pytorch:24.03-py3
The benchmarks can be reproduced with the provided scripts.
- includes baseline benches (e.g., standard fine-tuning, standard peft).
- benches for various acceleration sample configs.
See below CSV files for various results:
For deeper dive into details see framework/README.md.
IBM Research, Singapore
- Fabian Lim [email protected]
- Aaron Chew [email protected]
- Laura Wynter [email protected]