Comments (7)
They only share the same architecture, but not model weights due to different initialization. In practice, training a full-precision learner with distillation loss can indeed improve the performance. Below is a comparison on the classification accuracy of models trained with or without distillation loss, on CIFAR-10:
Model | Distillation? | Accuracy | Improvement |
---|---|---|---|
ResNet-20 | 91.93% | ||
ResNet-20 | + | 93.10% | 1.07% |
ResNet-32 | 92.59% | ||
ResNet-32 | + | 93.44% | 0.85% |
ResNet-44 | 92.76% | ||
ResNet-44 | + | 93.71% | 0.95% |
ResNet-56 | 93.23% | ||
ResNet-56 | + | 94.01% | 0.78% |
from pocketflow.
OK, thanks
from pocketflow.
The distill result is well when I distill model in small dataset, but when I distill resnet_v1_50
in ImageNet
, the result is not good. The top-5 of teacher model is 92.7%
, but top-5 of student model is 92%
.
from pocketflow.
- For ImageNet, we discover that distillation is sometimes helpful for training with compression algorithms, e.g. uniform quantization.
- It is also observed in some papers that simply using distillation on the classification logits can be harmful for training a full-precision model, e.g. Table 5 in [1]. Distilling with the attention map [2] or factor vector [1] may be a better choice.
[1] Jangho Kim, SeongUk Park, and Nojun Kwak, Paraphrasing Complex Network: Network Compression via Factor Transfer, NIPS 2018.
[2] Sergey Zagoruyko and Nikos Komodakis, Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer, ICLR 2017.
from pocketflow.
Thanks, I don't compress the model . I only want to finetune resnet_v1_50
to get high acuuracy. Can I use distillation.
from pocketflow.
Then you can try out the above two papers' algorithms (not implemented in PocketFlow).
- Attention Transfer's official implementation:
https://github.com/szagoruyko/attention-transfer
from pocketflow.
thanks
from pocketflow.
Related Issues (20)
- cifar10_channel pruned 的示例,通道剪枝(channel_pruning) 导出修改了计算图之后,速度比之前的更慢了! HOT 1
- Can the compression method provided by pocketflow be applied to MASK R-CNN? HOT 1
- QQ group HOT 1
- 我可以只用模型压缩部分么?
- TypeError: forward_train() missing 1 required positional argument: 'objects'
- Missing 1 required positional argument in constructor : data_format
- Download Pretrain Model But Get 502 Bad Gateway Error HOT 1
- You must feed a value for placeholder tensor 'model/input_1' with dtype float and shape [?,160,240,1]
- Question about export_chn_pruned_tflite_model.py HOT 1
- TF Version compatibility HOT 2
- Failed to create session
- Is it possible to compress the keras model with Pocket Flow
- Question about UniformLearner HOT 2
- Default tensorboard log output is huge
- FRCNN with VOC: Cannot batch tensors with different shapes in component 1.
- IndexError: list index out of range HOT 3
- Other issues:
- auto 通道裁剪问题
- test
- TF-Plus for Multi-GPU Training
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pocketflow.