Code for our paper:

"Unsupervised Attention Mechanism across Neural Network Layers"

by Baihan Lin (Columbia).

For the latest full paper: https://arxiv.org/abs/1902.10658

All the experimental results and analysis can be reproduced using the code in this repository. Feel free to contact me by [email protected] if you have any question about our work.

Abstract

Inspired by the adaptation phenomenon of neuronal firing, we propose an unsupervised attention mechanism (UAM) which computes the statistical regularity in the implicit space of neural networks under the Minimum Description Length (MDL) principle. Treating the neural network optimization process as a partially observable model selection problem, UAM constrained the implicit space by a normalization factor, the universal code length. We compute this universal code incrementally across neural network layers and demonstrated the flexibility to include data priors such as top-down attention and other oracle information. Empirically, our approach outperforms existing normalization methods in tackling limited, imbalanced and non-stationary input distribution in computer vision and reinforcement learning tasks. Lastly, UAM tracks dependency and critical learning stages across layers and recurrent time steps of deep networks.

Language: Python3, bash

Platform: MacOS, Linux, Windows

by Baihan Lin, Feb 2019

If you find this work helpful, please try the models out and cite our work. Thanks!

@article{lin2019unsupervised,
  title={{Unsupervised Attention Mechanism across Neural Network Layers}},
  author={Lin, Baihan},
  journal={arXiv preprint arXiv:1902.10658},
  year={2019}
}

An earlier version of the work was presented at the IJCAI 2019 Workshop on Human Brain and Artificial Intelligence in Macau, China. See the slides here (with only partial results in the arXiv above).

• Imbalanced MNIST task
• OpenAI gym's LunarLander-v2 game
• OpenAI gym's CarPole-v0 game

• Python 3
• PyTorch
• numpy and scikit-learn