Light

zkkli / psaq-vit Goto Github PK

View Code? Open in Web Editor NEW

121.0 3.0 13.0 487 KB

[ECCV 2022] Patch Similarity Aware Data-Free Quantization for Vision Transformers

License: Apache License 2.0

Python 100.00%

psaq-vit's Introduction

Patch Similarity Aware Data-Free Quantization for Vision Transformers

This repository contains the official PyTorch implementation for the ECCV 2022 paper "Patch Similarity Aware Data-Free Quantization for Vision Transformers". To the best of our knowledge, this is the first work on data-free quantization for vision transformers. Below are instructions for reproducing the results.

Installation

To install PSAQ-ViT and develop locally:

git clone https://github.com/zkkli/PSAQ-ViT.git
cd PSAQ-ViT

Quantization

You can quantize and evaluate a single model using the following command:

python test_quant.py [--model] [--dataset] [--w_bit] [--a_bit] [--mode]

optional arguments:
--model: Model architecture, the choises can be: 
         deit_tiny, deit_small, deit_base, swin_tiny, and swin_small.
--dataset: Path to ImageNet dataset.
--w_bit: Bit-precision of weights, default=8.
--a_bit: Bit-precision of activation, default=8.
--mode: Mode of calibration data,
        0: Generated fake data (PSAQ-ViT)
        1: Gaussian noise
        2: Real data

Example: Quantize DeiT-B with generated fake data (PSAQ-ViT).

python test_quant.py --model deit_base --dataset <YOUR_DATA_DIR> --mode 0

Example: Quantize DeiT-B with Gaussian noise.

python test_quant.py --model deit_base --dataset <YOUR_DATA_DIR> --mode 1

Example: Quantize DeiT-B with Real data.

python test_quant.py --model deit_base --dataset <YOUR_DATA_DIR> --mode 2

Results

Below are the experimental results of our proposed PSAQ-ViT that you should get on ImageNet dataset using an RTX 3090 GPU.

Model	Prec.	Top-1(%)	Prec.	Top-1(%)
DeiT-T (72.21)	W4/A8	65.57	W8/A8	71.56
DeiT-S (79.85)	W4/A8	73.23	W8/A8	76.92
DeiT-B (81.85)	W4/A8	77.05	W8/A8	79.10
Swin-T (81.35)	W4/A8	71.79	W8/A8	75.35
Swin-S (83.20)	W4/A8	75.14	W8/A8	76.64

Citation

We appreciate it if you would please cite the following paper if you found the implementation useful for your work:

@inproceedings{li2022psaqvit,
  title={Patch Similarity Aware Data-Free Quantization for Vision Transformers},
  author={Li, Zhikai and Ma, Liping and Chen, Mengjuan and Xiao, Junrui and Gu, Qingyi},
  booktitle={European Conference on Computer Vision},
  pages={154--170},
  year={2022}
}

psaq-vit's People

Contributors

Stargazers

Watchers

Forkers

hsvlab shadowclouds zkkkli bubu-1998 guangyanzhang dl-vit ironteen ther-nullptr chengquan jiangminhao husnejahan aramachandran2000 dr-usman1

psaq-vit's Issues

How long will the data generation take?

Thank you for sharing your brilliant work ~Could you share the information about your total training time of every stage, especially about the data generating time

The speed after quantization looks same as fp32

The final goal that we do quantization is to improve the inference speed. By moving the quantization module, I can get validation by using FP32 model. But I found the inference time is almost same as the time by using quantized model. Is that a right way to do the test? If not, how to get better inference speed after using quantized model?

Why we cannot use real data for calibrating quantization parameter instead?

If the target for generating fake data is to make the distribution to be same with the one from real data. Why we cannot directly collect the quantization parameter from the real data?

Could you plz provide the code for v2 of PSAQ

how

Failed to Reproduce the Data Generation Result of PSAQ-ViT

Hi Zhi kai,

Thank you very much for sharing the code of PSAQ-Vit and I have learned a lot from this excellent work. However, I encountered some questions about data generation technique proposed in your paper.

Specifically, followed the instruction in GitHub, I run the command python test_quant.py --model deit_base --dataset imagenet --mode 0, and then directly visualize the generated fake data. However, the generated data looks like random noisy and has a much difference to the results shown in your paper (Fig. 2.). I also attempted to adjust the learning rate and to optimize with more epochs but it still failed to improve the quality of fake data.

So could you provide me some suggestions about how to generate a more realistic data similar to yours? Any kindly reply will be helpful!

Best,
Yige

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.