By: Chenglei Si (River Valley High School)

XLNet has achieved impressive gains on RACE recently. You may refer to my other repo: https://github.com/NoviScl/XLNet_DREAM to see how to use XLNet for multiple-choice machine comprehension problems. Huggingface has updated their work pytorch_trainsformers, please refer to their repo for the documentation and more details of the new version.

This work is based on Pytorch implementation of BERT (https://github.com/huggingface/pytorch-pretrained-BERT). I adapted the original BERT model to work on multiple choice machine comprehension.

The code is tested with Python3.6 and Pytorch 1.0.0.

1. Download the dataset and unzip it. The default dataset directory is ./RACE
2. Run ./run.sh

I did some tuning and find the following hyperparameters to work reasonally well:

BERT_base: batch size: 32, learning rate: 5e-5, training epoch: 3

BERT_large: batch size: 8, learning rate: 1e-5 (DO NOT SET IT TOO LARGE), training epoch: 2

You can compare them with other results on the leaderboard.

BERT large achieves the current (Jan 2019) best result. Looking forward to new models that can beat BERT!

I have written a short report in this repo describing the details.