AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration [Paper]
Efficient and accurate low-bit weight quantization (INT3/4) for LLMs, supporting instruction-tuned models and multi-modal LMs.
The current release supports:
- AWQ search for accurate quantization.
- Pre-computed AWQ model zoo for LLMs (LLaMA-1&2, OPT, Vicuna, LLaVA; load to generate quantized weights).
- Memory-efficient 4-bit Linear in PyTorch.
- Efficient CUDA kernel implementation for fast inference (support context and decoding stage).
- Examples on 4-bit inference of an instruction-tuned model (Vicuna) and multi-modal LM (LLaVA).
Check out TinyChat, which delievers 2.3x faster inference performance for the LLaMA-2 chatbot on RTX 4090!
It also offers a turn-key solution for on-device inferecne of LLMs on resource-constrained edge platforms. With TinyChat, it is now possible to run large models on small and low-power devices even without Internet connection.
- [2023/07] ๐ฅ We released TinyChat, an efficient and lightweight chatbot interface based on AWQ. TinyChat enables efficient LLM inference on both cloud and edge GPUs. LLama-2-chat models are supported! Check out our implementation here.
- [2023/07] ๐ฅ We added AWQ support and pre-computed search results for Llama-2 models (7B & 13B). Checkout our model zoo here!
- [2023/07] We extended the support for more LLM models including MPT, Falcon, and BLOOM.
- Clone this repository and navigate to AWQ folder
git clone https://github.com/mit-han-lab/llm-awq
cd llm-awq
- Install Package
conda create -n awq python=3.10 -y
conda activate awq
pip install --upgrade pip # enable PEP 660 support
pip install -e .
-
For edge devices like Orin, before running the commands above, please:
- Modify pyproject.toml by commenting out this line.
- Manually install precompiled PyTorch binaries (>=2.0.0) from NVIDIA.
- Set the appropriate Python version for conda environment (e.g.,
conda create -n awq python=3.8 -y
for JetPack 5).
- Install efficient W4A16 (4-bit weight, 16-bit activation) CUDA kernel and optimized FP16 kernels (e.g. layernorm, positional encodings).
cd awq/kernels
python setup.py install
We provide pre-computed AWQ search results for multiple model families, including LLaMA, OPT, Vicuna, and LLaVA. To get the pre-computed AWQ search results, run:
# git lfs install # install git lfs if not already
git clone https://huggingface.co/datasets/mit-han-lab/awq-model-zoo awq_cache
The detailed support list:
Models | Sizes | INT4-g128 | INT3-g128 |
---|---|---|---|
LLaMA-2 | 7B/13B/70B | โ | โ |
LLaMA | 7B/13B/30B/65B | โ | โ |
Vicuna | 7B/13B | โ | |
MPT | 7B/30B | โ | |
Falcon | 7B/40B | โ | |
OPT | 125m/1.3B/2.7B/6.7B/13B/30B | โ | โ |
Bloom | 560m/3B/7B/ | โ | โ |
LLaVA-v0 | 13B | โ |
AWQ can be easily applied to various LMs thanks to its good generalization, including instruction-tuned models and multi-modal LMs. It provides an easy-to-use tool to reduce the serving cost of LLMs.
Here we provide two examples of AWQ application: Vicuna-7B (chatbot) and LLaVA-13B (visual reasoning) under ./examples
directory. AWQ can easily reduce the GPU memory of model serving and speed up token generation. It provides accurate quantization, providing reasoning outputs. You should be able to observe memory savings when running the models with 4-bit weights.
Note that we perform AWQ using only textual calibration data, depsite we are running on multi-modal input. Please refer to ./examples
for details.
We provide several sample script to run AWQ (please refer to ./scripts
). We use Vicuna 7B v1.5 as an example.
- Perform AWQ search and save search results
python -m awq.entry --entry_type search \
--model_path lmsys/vicuna-7b-v1.5 \
--search_path vicuna-7b-v1.5-awq
Note: if you use Falcon 7B, please pass --q_group_size 64
in order for it to work.
- Generate quantized weights and save them (INT4)
python -m awq.entry --entry_type quant \
--model_path lmsys/vicuna-7b-v1.5 \
--search_path vicuna-7b-v1.5-awq/awq_model_search_result.pt \
--quant_path vicuna-7b-v1.5-awq
- Load and evaluate the perplexity of the real quantized model weights (faster and uses less memory)
python -m awq.entry --entry_type perplexity \
--quant_path vicuna-7b-v1.5-awq \
--quant_file awq_model_w4_g128.pt
If you find AWQ useful or relevant to your research, please kindly cite our paper:
@article{lin2023awq,
title={AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration},
author={Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Dang, Xingyu and Han, Song},
journal={arXiv},
year={2023}
}
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers