Note: This code is not up-to-date, please refer to the implementation by the original authors: https://github.com/kentonl/ran

This is a PyTorch implementation of Recurrent Additive Networks (RAN) by Kenton Lee, Omer Levy, and Luke Zettlemoyer:

http://www.kentonl.com/pub/llz.2017.pdf

The RAN model is implemented in ran.py.

Code for running Penn Tree Bank (PTB) experiments is taken from:

https://github.com/pytorch/examples/tree/master/word_language_model

To run PTB experiments, clone this repository:

git clone https://github.com/bheinzerling/ran

and then do:

cd ran
python main.py --cuda --emsize 256 --nhid 1024 --dropout 0.5 --epochs 100 --nlayers 1 --batch-size 512 --model RAN

This should result in a test set perplexity which roughly agrees with the RAN (tanh) result reported in the paper:

End of training | test loss  4.78 | test ppl   119.40

Better results can be achieved with smaller batch sizes, e.g. with batch size 40:

End of training | test loss  4.45 | test ppl    85.24

batch size 20:

| End of training | test loss  4.42 | test ppl    83.42

batch size 10:

| End of training | test loss  4.41 | test ppl    82.62

batch size 5:

| End of training | test loss  4.49 | test ppl    89.21