Barebones transformer decoder with adaptive input and adaptive softmax.

The model is largely based on the pytorch language model example code. The transformer is trained on the wikitext-2 dataset by default.

python main.py --cuda --epochs 6 --adsmax   # Train with adaptive softmax
python main.py --cuda --epochs 6 --adinp    # Train with adaptive input
python main.py --cuda --epochs 6 --tied_weights # Train with adaptive input and softmax with tied weights

Wikitext-2 results for 6 training epochs

Text8 results for 10 training epochs, 512 dimension hidden and embedding layer