distill model in full-prec mode about pocketflow HOT 7 CLOSED

They only share the same architecture, but not model weights due to different initialization. In practice, training a full-precision learner with distillation loss can indeed improve the performance. Below is a comparison on the classification accuracy of models trained with or without distillation loss, on CIFAR-10:

from pocketflow.

OK, thanks

from pocketflow.

The distill result is well when I distill model in small dataset, but when I distill resnet_v1_50 in ImageNet, the result is not good. The top-5 of teacher model is 92.7% , but top-5 of student model is 92%.

from pocketflow.

1. For ImageNet, we discover that distillation is sometimes helpful for training with compression algorithms, e.g. uniform quantization.
2. It is also observed in some papers that simply using distillation on the classification logits can be harmful for training a full-precision model, e.g. Table 5 in [1]. Distilling with the attention map [2] or factor vector [1] may be a better choice.

[1] Jangho Kim, SeongUk Park, and Nojun Kwak, Paraphrasing Complex Network: Network Compression via Factor Transfer, NIPS 2018.
[2] Sergey Zagoruyko and Nikos Komodakis, Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer, ICLR 2017.

from pocketflow.

Thanks, I don't compress the model . I only want to finetune resnet_v1_50 to get high acuuracy. Can I use distillation.

from pocketflow.

Then you can try out the above two papers' algorithms (not implemented in PocketFlow).

• Attention Transfer's official implementation:
  https://github.com/szagoruyko/attention-transfer

from pocketflow.

thanks

from pocketflow.