This repository contains the source code for the EMNLP 2022 (Findings) paper: Task Compass: Scaling Multi-task Pre-training with Task Prefix [PDF]. In this paper, we propose a task prefix guided multi-task pre-training framework (CompassMTL) to explore the relationships among tasks. CompassMTL is based on the DeBERTa architecture, trained with 40 natural language understanding tasks. Please refer more details in our paper.

• numpy
• torch
• transformers==4.17.0
• wandb
• sentencepiece
• sklearn
• datasets

Download data from datasets

Training:

bash run_train.sh

evaluate:

bash run_evaluate.sh

We provide the models and outputs for the ANLI and HellaSwag Commonsense Reasoning tasks:

Our sinlge models for ANLI and HellaSwag are available at reasoning_models

The outputs can be found at model_outputs.

Please kindly cite this paper in your publications if it helps your research:

@inproceedings{zhang2022task,
  title={Task Compass: Scaling Multi-task Pre-training with Task Prefix},
  author={Zhang, Zhuosheng and Wang, Shuohang and Xu, Yichong and Fang, Yuwei and Yu, Wenhao and Liu, Yang and Zhao, Hai and Zhu, Chenguang and Zeng, Michael},
  booktitle={arXiv preprint arXiv:2210.06277},
  year={2022}
}