This repository contains the code, data and model for the paper "CoUDA: Coherence Evaluation via Unified Data Augmentation" (NAACL 2024)

To replicate our results, follow these steps to download the code and necessary dependencies:

git clone https://github.com/dwzhu-pku/CoUDA.git
cd CoUDA
pip install -r requirements.txt

The datasets used for training and evaluation can be found under data/train and data/eval, respectively.

For our trained model, please download it via this link, and place it under models/couda_model. Note that the results reported in the paper is averaged across 3 runs with different random seeds. The uploaded one is selected in random, which turns out to have slightly lower score.

To calculate correlation scores with human ratings, you may run:

bash scripts/run_coh_main.sh

To calcuate pair-wise ranking scores, you may run:

bash scripts/run_eval_pair.sh

If you find this repo helpful, please cite our paper as follows: