princetonvisualai / rememberthepast-datasetdistillation Goto Github PK

Hi.

This is an inspiring work about dataset distillation. The training pipeline (Algorithm 1 in the paper) aims to maximize the performance of the model that

• trained by synthetic data D_s (Algorithm lines 8-13)
• evaluated by real data D_{tr} (Algorithm line 14).

This pipeline is similar to Dataset Distilation[1], which also aims to maximize the performance trained by S and evaluated by T. However, [1] raises a problem that the initial parameter $\theta_0$ (Algorithm 1 line 6) significantly influences the performance of the whole algorithm. This is because the final parameter $\theta_T$ is dependent on the initial parameter $\theta_0$ and thus the loss function $J(\phi)$ and optimization direction OPT-STEP $(\phi, J(\phi), \alpha_1, \beta_1)$ is dependent on the initial parameter $\theta_0$.

So, I'm confusing how this work deals with this initial parameter problem.

Thanks!~

[1]Wang, T., Zhu, J., Torralba, A., & Efros, A. A. (2018). Dataset Distillation. ArXiv. https://doi.org/10.48550/arXiv.1811.10959

Hello,

This is a great work! We are trying to follow up, could you please provide some pretrained distilled datasets, like the ones here?

Many thanks!

Hi.

How to visualize the synthetic images if I activate ZCA pre-processing (as in Figure 7-9 in your paper)?

I tried to visualize the images stored in the weight in the module nn.Embedding using BPTT without memory addressing, but it appears to be an almost black image. Is this a normal case for the BPTT+momentum algorithm? I got ~49% accuracy on ipc1 in CIFAR10, which is a close value to the reported result in your paper. So I guess maybe I miss something about how to visualize the images. Perhaps in the case of ZCA?

Thanks.

The scripts for Back-propagation through time (BPTT) with memory addressing needs "compressors_bptt_test.yml"