Clone this repository:
git clone https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation.git
Build a new environment and install:
pip install -r requirements.txt
CIFAR100, CIFAR10 and Tiny_ImageNet; For CIFAR100 and CIFAR10, our codes will download the datasets automatically. For Tiny-ImageNet, you should download and put in the dir: "data/". The follow instruction and commands are for CIFAR100.
You can skip this step by using our pre-trained models in here. Download and unzip to: experiments/pretrained_teacher_models/
Use ''--model_dir'' to specify the directory of "parameters", model saving and log saving.
For example, normally train ResNet18 to obtain the pre-trained teacher:
CUDA_VISIBLE_DEVICES=0 python main.py --model_dir experiments/base_experiments/base_resnet18/
We ignore the command ''CUDA_VISIBLE_DEVICES=gpu_id'' in the following commands
Normally train MobileNetV2 to obtain the baseline model and baseline accuracy:
python main.py --model_dir experiments/base_experiments/base_mobilenetv2/
Normally train ResNeXt29 to obtain the baseline model and baseline accuracy:
python main.py --model_dir experiments/base_experiments/base_resnext29/
The baseline accuracy (in %) on CIFAR100 is:
Model | Baseline Acc |
---|---|
MobileNetV2 | 68.38 |
ShuffleNetV2 | 70.34 |
ResNet18 | 75.87 |
ResNet50 | 78.16 |
GoogLeNet | 78.72 |
Desenet121 | 79.04 |
ResNeXt29 | 81.03 |
Normal KD: ResNet18 teach MobileNetV2
python main.py --model_dir experiments/kd_experiments/mobilenet_distill/resnet18_teacher/
If you find this repo useful, please consider citing:
@inproceedings{yuan2020revisiting,
title={Revisiting Knowledge Distillation via Label Smoothing Regularization},
author={Yuan, Li and Tay, Francis EH and Li, Guilin and Wang, Tao and Feng, Jiashi},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={3903--3911},
year={2020}
}