Code for the paper "Structured Prediction as Translation between Augmented Natural Languages" (ICLR 2021).
If you use this code, please cite the paper using the bibtex reference below.
@inproceedings{tanl,
title={Structured Prediction as Translation between Augmented Natural Languages},
author={Giovanni Paolini and Ben Athiwaratkun and Jason Krone and Jie Ma and Alessandro Achille and Rishita Anubhai and Cicero Nogueira dos Santos and Bing Xiang and Stefano Soatto},
booktitle={9th International Conference on Learning Representations, {ICLR} 2021},
year={2021},
}
- Python 3.6+
- PyTorch (tested with version 1.7.1)
- Transformers (tested with version 4.0.0)
- NetworkX (tested with version 2.5, only used in coreference resolution)
You can install all required Python packages with pip install -r requirements.txt
By default, datasets are expected to be in data/DATASET_NAME
.
Dataset-specific code is in datasets.py.
For example, the CoNLL04 and ADE datasets (joint entity and relation extraction) in the correct format can be downloaded using https://github.com/markus-eberts/spert/blob/master/scripts/fetch_datasets.sh. For other datasets, pre-processing and links are documented in the code.
Use the following command:
python run.py JOB
The JOB
argument refers to a section of the config file, which by default is config.ini
.
A sample config file is provided, with settings that allow for a faster training and less memory usage than the settings used to obtain the final results in the paper.
For example, to replicate the paper's results on CoNLL04, have the following section in the config file:
[conll04_final]
datasets = conll04
model_name_or_path = t5-base
num_train_epochs = 200
max_seq_length = 256
max_seq_length_eval = 512
train_split = train,dev
per_device_train_batch_size = 8
per_device_eval_batch_size = 16
do_train = True
do_eval = False
do_predict = True
episodes = 1-10
num_beams = 8
Then run python run.py conll04_final
.
Note that the final results will differ slightly from the ones reported in the paper, due to small code changes and randomness.
Config arguments can be overwritten by command line arguments.
For example: python run.py conll04_final --num_train_epochs 50
.
If do_train = True
, the model is trained on the given train split (e.g., 'train'
) of the given datasets.
The final weights and intermediate checkpoints are written in a directory such as experiments/conll04_final-t5-base-ep200-len256-b8-train
, with one subdirectory per episode.
Results in JSON format are also going to be saved there.
In every episode, the model is trained on a different (random) permutation of the training set. The random seed is given by the episode number, so that every episode always produces the same exact model.
Once a model is trained, it is possible to evaluate it without training again.
For this, set do_train = False
or (more easily) provide the -e
command-line argument: python run.py conll04_final -e
.
If do_eval = True
, the model is evaluated on the 'dev'
split.
If do_predict = True
, the model is evaluated on the 'test'
split.
The following are the most important command-line arguments for the run.py
script.
Run python run.py -h
for the full list.
-c CONFIG_FILE
: specify config file to use (default isconfig.ini
)-e
: only run evaluation (overwrites the settingdo_train
in the config file)-a
: evaluate also intermediate checkpoints, in addition to the final model-v
: print results for each evaluation run-g GPU
: specify which GPU to use for evaluation
The following are the most important arguments for the config file. See the sample config file to understand the format.
datasets
(str): comma-separated list of datasets for trainingeval_datasets
(str): comma-separated list of datasets for evaluation (default is the same as for training)model_name_or_path
(str): path to pretrained model or model identifier from huggingface.co/models (e.g.t5-base
)do_train
(bool): whether to run training (default is False)do_eval
(bool): whether to run evaluation on thedev
set (default is False)do_predict
(bool): whether to run evaluation on thetest
set (default is False)train_split
(str): comma-separated list of data splits for training (default istrain
)num_train_epochs
(int): number of train epochslearning_rate
(float): initial learning rate (default is 5e-4)train_subset
(float > 0 and <=1): portion of training data to effectively use during training (default is 1, i.e., use all training data)per_device_train_batch_size
(int): batch size per GPU during training (default is 8)per_device_eval_batch_size
(int): batch size during evaluation (default is 8; only one GPU is used for evaluation)max_seq_length
(int): maximum input sequence length after tokenization; longer sequences are truncatedmax_output_seq_length
(int): maximum output sequence length (default ismax_seq_length
)max_seq_length_eval
(int): maximum input sequence length for evaluation (default ismax_seq_length
)max_output_seq_length_eval
(int): maximum output sequence length for evaluation (default ismax_output_seq_length
ormax_seq_length_eval
ormax_seq_length
)episodes
(str): episodes to run (default is0
; an interval can be specified, such as1-4
; the episode number is used as the random seed)num_beams
(int): number of beams for beam search during generation (default is 1)multitask
(bool): if True, the name of the dataset is prepended to each input sentence (default is False)
See arguments.py and transformers.TrainingArguments for additional config arguments.