In this study, we extend our previous work Studying the usage of text-to-text transfer transformer for code-related tasks paying particular attention at the role played by pre-training and multi-task fine-tuning on the model's performance.

In order to pre-train and then finetune a T5 small model, we need a new sentencepiece model to accommodate the expanded vocabulary given by the java programming language, abstracted java tokens, and technical natural language.

• How to train a new SPmodel

  Pythonic way

  pip install sentencepiece==0.1.96
  import sentencepiece as spm
  spm.SentencePieceTrainer.train('--input=pretraining.txt --model_prefix=dl4se --vocab_size=32000 --bos_id=-1  --eos_id=1 --unk_id=2 --pad_id=0') 
  

  The new SPmodel has to be trained on the entire pre-training corpus. The tokenizer we trained is available here

• Set up a GCS Bucket 💡

  To Set up a new GCS Bucket for training and fine-tuning a T5 Model, please follow the orignal guide provided by Google . Here the link: https://cloud.google.com/storage/docs/quickstart-console Subsequently, by following the jupyter notebook we provide for pre-train and fine-tune the network, you should be able to set up the final environment.

• Datasets 📎

  The datasets for the pre-training and the fine-tuning can be found here

• Pre-trainig/Fine-tuning 💻

  To pre-train and then, fine-tune T5, please use the script we provide here:

  • Pre-Training
  • Fine-Tuning

• How to generate the predictions 📈

  First you need to convert the TF model into a pytorch model by using TF_to_Pytorch , then run Generate Results

• Models 📊

  • No Pre-Training
  • Pre-Training Single Task
  • Pre-Training Multi Task (Proportional Sampling)
  • Pre-Training Multi Task (Balanced Sampling)

• Predictions: 📂 Click Me!

• Additional: 📋

  Under Miscellaneous, you can find the additional script used for computing the statistical tests, the complementary analysis, the overlap and data snooping analysis.

• Extra: 📋

  • Hyperparameters tuning results are available here
  • To navigate the replication package click here