The code for ICML'17 paper on Variational Dropout Sparsifies Deep Neural Networks (slides). We showed that Variational Dropout leads to extremely sparse solutions both in fully-connected and convolutional layers. The number of parameters was reduced up to 280 times on LeNet architectures and up to 68 times on VGG-like networks with a negligible decrease of accuracy. This effect is similar to automatic relevance determination effect, but prior distribution is fixed, so there is no additional overfitting risk.

We visualize weights of Sparse VD LeNet-5-Caffe network and demonstrate several filters of the first convolutional layer and the piece of a fully-connected layer :)

Comparison of different sparsity-inducing techniques (Pruning (Han et al., 2015b;a), DNS (Guo et al., 2016), SWS (Ullrich et al., 2017)) on LeNet architectures. Our method provides the highest level of sparsity with a similar accuracy

Accuracy and sparsity level for VGG-like architectures of different sizes. The number of neurons and filters scales as k. Dense networks were trained with Binary Dropout, and Sparse VD networks were trained with Sparse Variational Dropout on all layers. The overall sparsity level, achieved by our method, is reported as a dashed line. The accuracy drop is negligible in most cases, and the sparsity level is high, especially in larger networks.

sudo apt install virtualenv python-pip python-dev
virtualenv venv --system-site-packages
source venv/bin/activate

pip install numpy tabulate 'ipython[all]' sklearn matplotlib seaborn  
pip install --upgrade https://github.com/Theano/Theano/archive/master.zip
pip install --upgrade https://github.com/Lasagne/Lasagne/archive/master.zip

source ~/venv/bin/activate
cd variational-dropout-sparsifies-dnn
THEANO_FLAGS='floatX=float32,device=gpu0,lib.cnmem=1' ipython ./experiments/<experiment>.py

If you found this code useful please cite our paper

@article{molchanov2017vparsevd,
  title={Variational Dropout Sparsifies Deep Neural Networks},
  author={Molchanov, Dmitry and Ashukha, Arsenii and Vetrov, Dmitry},
  journal={arXiv preprint arXiv:1701.05369},
  year={2017}
}