IMM incrementally matches the moment of the posterior distribution of neural networks, which is trained for the first and the second task respectively. The experiment is only for the shuffled MNIST task.
If you use this code as part of any published research, we'd really appreciate it if you could cite the following paper:
@inproceedings{Lee2017,
author = {Lee, Sang-Woo Lee and Kim, Jin-Hwa and Jun, Jaehyun and Ha, Jung-Woo and Zhang, Byoung-Tak},
booktitle = {Advances In Neural Information Processing Systems 30},
title = {{Overcoming Catastrophic Forgetting by Incremental Moment Matching (IMM)}},
year = {2017}
}
Tested at
- Python v3.5
- Tensorflow v1.4.0
Shuffled MNIST data was used for the experiment with three sequential tasks. The first dataset is the same as the original MNIST dataset. In the second and third tasks, the input pixels of all images are shuffled with a fixed, random permutation.
no need to download mnist dataset. The download code is included.
By default, main codes is executed with mean-imm and mode-imm. But, the main codes of SGD, L2-transfer, Drop-transfer or L2&drop-transfer are separated for its clear code.
$ python main.py
$ python main_l2.py
$ python main_drop.py
$ python main_l2_drop.py
$ python main_l2_drop.py --dropout 0.5 --learning_rate 0.01 --alpha 0.5 --epoch 100
--mean_imm # include Mean-IMM
--mode_imm # include Mode-IMM
--dropout [DROPOUT_RATE] # dropout rate of hidden layers
--alpha [ALPHA] # alpha(K) of Mean & Mode IMM (cf. equation (3)~(8) in the article)
--epoch [EPOCH] # the number of training epoch
--optimizer [OPTIMIZER] # the method name of optimization. (SGD, Adam or Momentum)
--learning_rate [RATE] # learning rate of optimizer
--batch_size [BATCH_SIZE] # mini batch size
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent