Code Monkey home page Code Monkey logo

squad's Introduction

Setup

  1. Make sure you have Miniconda installed

    1. Conda is a package manager that sandboxes your project’s dependencies in a virtual environment
    2. Miniconda contains Conda and its dependencies with no extra packages by default (as opposed to Anaconda, which installs some extra packages)

  2. cd into src, run conda env create -f environment.yml

    1. This creates a Conda environment called squad

  3. Run conda activate squad

    1. This activates the squad environment
    2. Do this each time you want to write/test your code

  4. Run python setup.py

    1. This downloads SQuAD 2.0 training and dev sets, as well as the GloVe 300-dimensional word vectors (840B)
    2. This also pre-processes the dataset for efficient data loading
    3. For a MacBook Pro on the Stanford network, setup.py takes around 30 minutes total

  5. Browse the code in train.py

    1. The train.py script is the entry point for training a model. It reads command-line arguments, loads the SQuAD dataset, and trains a model.
    2. You may find it helpful to browse the arguments provided by the starter code. Either look directly at the parser.add_argument lines in the source code, or run python train.py -h.

Experiments

Id Code Description Notes load_path train-NLL train-F1 train-EM train-AvNA dev-NLL dev-F1 dev-EM dev-AvNA
baseline models.py Baseline No hyper-params tuning, hence dev-set used like test set. save/train/baseline-05/best.pth.tar 1.73 76.40 68.82 85.71 3.35 57.31 53.87 64.44
baseline_plus_1 models.py Baseline with larger modelsize and number of layers. --hidden_size 150 , --drop_prob 0.3 - The model appears to have a higher bias than baseline although the bigger capacity. This is due to the higher dropout probability. save/train/baseline_plus_1-01/best.pth.tar 02.27 68.44 60.75 80.29 03.22 56.62 53.25 63.75
baseline_char_embed models.py Baseline with character embedding. The model comes with higher capacity and overfits after 3 epochs. It is necessary some regularization. save/train/baseline_char_embed-04/best.pth.tar 05.27 52.19 52.19 52.14
baseline_char_embed models.py Baseline with character embedding and higher dropout (0.3 vs. 0.2). --drop_prob 0.3 save/train/baseline_char_embed-05/best.pth.tar 01.69 76.95 69.09 86.17 03.37 58.02 54.34 65.33
baseline_char_embed models.py Baseline with character embedding and higher dropout (0.4 vs. 0.2). --drop_prob 0.4 save/train/baseline_char_embed-06/best.pth.tar 02.31 67.67 59.99 79.78 03.07 57.13 53.79 64.68
baseline_char_embed_bert_enc models.py Baseline with character embedding and higher dropout (0.3). --drop_prob 0.3 and BERT-encoder --hidden_size 256 enstead of RNN-encoder save/train/baseline_char_embed_bert_enc-19/best.pth.tar 01.59 78.12 70.96 86.72 03.36 58.89 55.82 65.60
baseline_char_embed_bert_enc models.py Baseline with character embedding, dropout (0.2) leaving BERT encoder dropout to 0.1 --drop_prob 0.2 and BERT-encoder --hidden_size 256 instead of RNN-encoder 89.95-60.30=~30 --> low model bias, high variance ... there is material to work on save/train/baseline_char_embed_bert_enc-18/best.pth.tar 00.84 89.95 83.69 94.87 03.65 60.30 57.02 67.72
baseline_char_embed_bert_enc models.py Baseline with character embedding, dropout (0.2) leaving BERT encoder dropout to 0.1 --drop_prob 0.2 and BERT-encoder --hidden_size 256 instead of RNN-encoder. BERT encoder has 3 layers (instead of 6) and d_ff, the depth of feed-forward layer, is ~depth of embedding (instead of 2048) 73.98-60.41=~13 save/train/baseline_char_embed_bert_enc-17/best.pth.tar 01.92 73.98 66.93 83.74 02.89 60.41 57.39 66.80
baseline_char_embed_bert_enc models.py Baseline with character embedding, dropout (0.3) leaving BERT encoder dropout to 0.1 --drop_prob 0.3 and BERT-encoder --hidden_size 256 instead of RNN-encoder save/train/baseline_char_embed_bert_enc-20/best.pth.tar 01.44 80.66 73.53 88.42 03.19 60.03 56.86 66.85
baseline_char_embed_bert_enc models.py Baseline with character embedding, dropout (0.2) leaving BERT encoder dropout to 0.1 --drop_prob 0.2 and BERT-encoder --hidden_size 256 instead of RNN-encoder. L2 regularization --l2_wd 0.1 L2 regularization & dropout don't look to work well togther. save/train/baseline_char_embed_bert_enc-22/best.pth.tar 06.80 33.37 33.37 33.36 05.70 52.19 52.19 52.14
baseline_char_embed_bert_enc models.py Baseline with character embedding, dropout (0.2) leaving BERT encoder dropout to 0.1 --drop_prob 0.2 and BERT-encoder --hidden_size 256 instead of RNN-encoder. BERT encoder has 3 layers (instead of 6) and d_ff, the depth of feed-forward layer, is ~depth of embedding (instead of 2048). + twist embeddings save/train/baseline_char_embed_bert_enc-27/best.pth.tar 01.63 78.05 70.89 86.54 03.12 59.88 56.80 66.64
baseline_char_embed_bert_enc_bert_mod models.py Baseline with character embedding, dropout (0.2) leaving BERT encoder dropout to 0.1 --drop_prob 0.2 and BERT-encoder --hidden_size 256 instead of RNN-encoder with 4 layers. Model layer is a BERT encoder as well (6 layers). Necessary --batch_size 16 prehaps, is the capacity of the network too high? save/train/baseline_char_embed_bert_enc_bert_mod-05/best.pth.tar 06.59 33.37 33.37 33.36 05.42 52.19 52.19 52.14
baseline_char_embed_bert_enc_bert_mod models.py Baseline with character embedding, dropout (0.2) leaving BERT encoder dropout to 0.1 --drop_prob 0.2 and BERT-encoder --hidden_size 256 instead of RNN-encoder. BERT encoder has 3 layers (instead of 6) and d_ff, the depth of feed-forward layer, is ~depth of embedding (instead of 2048). Modeling layer comes with another BERT encoder (3 layer). --batch_size 32 save/train/baseline_char_embed_bert_enc_bert_mod-07/best.pth.tar 02.88 67.81 59.19 81.80 04.45 53.49 49.22 61.92

squad's People

Contributors

chrischute avatar ginoteseianthem avatar minggg avatar rohitmusti avatar

Watchers

avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.