Create and activate a conda environment with

conda env create -f environment.yml
conda activate outliersvsfreq

To replicate the fine tuning of multiberts checkpoints move to the multiberts folder and run the download script this will take some disk space (~15gb or more):

bash donwload_and_convert_to_pt_multiberts_seed_1.sh

then, go back to the main directory and run:

bash run_multiberts.sh

To replicate the results on VIT run:

bash run_vit.sh

To replicate the remaining results in the paper first run:

bash run_glue.sh bert-base-uncased

NOTE: This runs all the fine-tuning for bert-base-uncased and all the experiment zeroing outliers out, it takes a long time to run.

Then run all experiments in the paper except the bert-medium pretraining use

bash run_experiments.sh

inspect the file to see which experiments are run.

All the plots in the paper should be replicable using the paper_plots.ipynb notebook

Finally to run bert_medium pretraining the following should run the SPLIT case

NOTE: adjust the checkpointing using huggingface settings as this is too variable to be fixed for all machines.

cd pre_training_bert_medium
python -m outliersvsfreq.experiments.pretrain_bert_medium \
    --output_dir mlm_run \
    --preprocessing_num_workers 8 \
    --do_train true \
    --per_device_train_batch_size 128 \
    --per_device_eval_batch_size 128 \
    --warmup_step 30000 \
    --weight_decay 0.01 \
    --num_train_epochs 4 \
    --do_eval true \
    --validation_split_percentage 1 \
    --learning_rate 1.e-4 \
    --max_seq_length 256 \
    --do_split_in_sentences true \
    --is_test false \
    --randomize_tokens false \
    --few_special_tokens false \

changing the last two flags runs the different data preparation.

For RANDOMIZE_TOKENS they should be set to

    --randomize_tokens true
    --few_special_tokens false

and for ONE_SEP to

    --randomize_tokens false
    --few_special_tokens true

NOTE: You may need to install a model for Spacy to work

python -m spacy download "en_core_web_sm"