How long did it take you to train this model?

Early solution for Google AI4Code competition

Overview

This solution is based on Amet Erdem's baseline . Instead of predicting the cell position with only the markdown itself, we randomly sample up to 20 code cells to act as the global context. So your input will look something like this: <s> Markdown content <s> Code content 1 <s> Code content 2 <s> ... <s> Code content 20 <s>

Ez pz.

Preprocessing

To extract features for training, including the markdown-only dataframes and sampling the code cells needed for each note book, simply run:

$ python preprocess.py

Your outputs will be in the ./data folder:

project
│   train_mark.csv
│   train_fts.json   
|   train.csv
│   val_mark.csv
│   val_fts.json
│   val.csv

Training

I found codebert-base to be the best of all the transformers:

$ python train.py --md_max_len 64 --total_max_len 512 --batch_size 8 --accumulation_steps 4 --epochs 5 --n_workers 8

The validation scores should read 0.84+ after 3 epochs, and also correlates well with the public LB.

Inference

Please refer to my public notebook: https://www.kaggle.com/code/suicaokhoailang/stronger-baseline-with-code-cells

suicao / ai4code-baseline Goto Github PK

ai4code-baseline's Introduction

Early solution for Google AI4Code competition

Overview

Preprocessing

Training

Inference

ai4code-baseline's People

Contributors

Stargazers

Watchers

Forkers

ai4code-baseline's Issues