This monorepo collects libraries of packages that accelerate fine-tuning / training of large models, intended to be part of the fms-hf-tuning suite.

This package is in BETA under extensive development. Expect breaking changes!

This is intended to be a collection of many acceleration routines (including accelerated peft and other techniques). Below demonstrates a concrete example to show how to accelerate your tuning experience with tuning/sft_trainer.py from fms-hf-tuning.

Below instructions for accelerated peft fine-tuning. In particular GPTQ-LoRA tuning with the AutoGPTQ triton_v2 kernel; this kernel is state-of-the-art provided by jeromeku on Mar 2024:

1. Checkout fms-hf-tuning and install the framework library:

   pip install -e .[fms-accel]
   

   or alternatively install the framework directly:

   pip install git+https://github.com/foundation-model-stack/fms-acceleration.git#subdirectory=plugins/framework
   

2. The above installs the command line utility fms_acceleration.cli, which can then be used to install plugins. Use list to view available plugins; this list updates as more plugins get developed:

   $ python -m fms_acceleration.cli list
   
   Choose from the list of plugin shortnames, and do:
   * 'python -m fms_acceleration.cli install <pip-install-flags> PLUGIN_NAME'.
   
   List of PLUGIN_NAME [PLUGIN_SHORTNAME]:
   
   1. fms_acceleration_peft [peft]
   

   and then install the plugin. We install the fms-acceleration-peft plugin for GPTQ-LoRA tuning with triton v2 as:

   python -m fms_acceleration.cli install fms_acceleration_peft
   

   The above is the equivalent of:

   pip install git+https://github.com/foundation-model-stack/fms-acceleration.git#subdirectory=plugins/accelerated-peft
   

3. Prepare a YAML configuration for the acceleration framework plugins
   • Full sample configuration list shows the plugins required for the configs.
   • Accelerated GPTQ-LoRA configuration here.
   • Once fms-acceleration-peft plugin was installed (recall above), the framework uses the plugins given the framework configuration; for more details see framework/README.md.
4. Run sft_trainer.py with the following arguments:

   • --acceleration_framework_config_file pointing to framework configuration YAML.

   • arguments required for correct operation (e.g., if using accelerated peft, then peft_method is required).

     • For sample configurations, these arguments can be referred from defaults.yaml and scenarios.yaml.
     • Arguments not critical to the plugins found in defaults.yaml. These can be taken with liberty.
     • Arguments critcal to plugins found in scenarios.yaml. The relevant section of scenarios.yaml, is the one whose framework_config entries, match the shortname of the sample configuration of interest.

   • More info on defaults.yaml and scenarios.yaml found here.

     # when using sample-configurations, arguments can be referred from
     # defaults.yaml and scenarios.yaml
     python sft_trainer.py \
         --acceleration_framework_config_file sample-configurations/acceleration_framework.yaml \
         ... # arguments from default.yaml
         ...  # arguments from scenarios.yaml
     

Over time, more plugins will be updated, so please check here for the latest accelerations!.

This repo requires CUDA to compute the kernels, and it is convinient to use NVidia Pytorch Containers that already comets with CUDA installed. We have tested with the following versions:

• pytorch:24.03-py3

The benchmarks can be reproduced with the provided scripts.

• includes baseline benches (e.g., standard fine-tuning, standard peft).
• benches for various acceleration sample configs.

See below CSV files for various results:

• A100-80GB.
• L40-40GB.

For deeper dive into details see framework/README.md.

IBM Research, Singapore

• Fabian Lim [email protected]
• Aaron Chew [email protected]
• Laura Wynter [email protected]