The official PyTorch implementation of our NeurIPS 2023 poster paper:

Sparse Parameterization for Epitomic Dataset Distillation

Xing Wei, Anjia Cao, Funing Yang, and Zhiheng Ma.

GitHub maintainer: Anjia Cao

The success of deep learning relies heavily on large and diverse datasets, but the storage, preprocessing, and training of such data present significant challenges. To address these challenges, dataset distillation techniques have been proposed to obtain smaller synthetic datasets that capture the essential information of the originals. In this paper, we introduce a Sparse Parameterization for Epitomic datasEt Distillation (SPEED) framework, which leverages the concept of dictionary learning and sparse coding to distill epitomes that represent pivotal information of the dataset. SPEED prioritizes proper parameterization of the synthetic dataset and introduces techniques to capture spatial redundancy within and between synthetic images. We propose Spatial-Agnostic Epitomic Tokens (SAETs) and Sparse Coding Matrices (SCMs) to efficiently represent and select significant features. Additionally, we build a Feature-Recurrent Network (FReeNet) to generate hierarchical features with high compression and storage efficiency. Experimental results demonstrate the superiority of SPEED in handling high-resolution datasets, achieving state-of-the-art performance on multiple benchmarks and downstream applications. Our framework is compatible with a variety of dataset matching approaches, enhancing their performance. This work highlights the importance of proper parameterization in epitomic dataset distillation and opens avenues for efficient representation learning.

• Standard Benchmarks (CIFAR10, CIFAR100, and TinyImageNet):

• High Resolution Datasets (ImageNet subsets):

• Clone the repo:

git clone https://github.com/MIV-XJTU/SPEED.git
cd SPEED

• Create a new environment:

conda create -n speed python==3.8
conda activate speed
pip install -r requirements.txt

• For CIFAR10 and CIFAR100:

python buffer.py --dataset {CIFAR10/CIFAR100} --model ConvNet --train_epochs 50 --num_experts 100 --zca
python distill.py --dataset {CIFAR10/CIFAR100} --model ConvNet --zca

• For TinyImageNet:

python buffer.py --dataset Tiny --model ConvNetD4 --train_epochs 50 --num_experts 100
python distill.py --dataset Tiny --model ConvNetD4

• For ImageNet subsets (e.g. ImageNette):

python buffer.py --dataset ImageNet --subset imagenette --model ConvNetD5 --train_epochs 50 --num_experts 100
python distill.py --dataset ImageNet --subset imagenette --model ConvNetD5

More hyperparameter settings are concluded at the end of networks.py. After distillation, you will obtain four components: saet, scm, freenet, and syn_lr.

• For CIFAR10 and CIFAR100:

python eval.py --dataset {CIFAR10/CIFAR100} --model ConvNet --zca

• For TinyImageNet:

python eval.py --dataset Tiny --model ConvNetD4

• For ImageNet subsets (e.g. ImageNette):

python eval.py --dataset ImageNet --subset imagenette --model ConvNetD5

Before evaluation, please specify the distilled components' paths: --saet_path, --scm_path, --freenet_path, --syn_lr_path.

Our work is implemented base on the following projects. We really appreciate their excellent open-source works!

• DatasetCondensation [related paper][related paper]
• mtt-distillation [related paper]
• DatasetFactorization [related paper]

If any parts of our paper and code help your research, please consider citing us and giving a star to our repository.

@inproceedings{wei2023sparse,
    title={Sparse Parameterization for Epitomic Dataset Distillation},
    author={Wei, Xing and Cao, Anjia and Yang, Funing and Ma, Zhiheng},
    booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
    year={2023}
}