POQue is a crowdsourced dataset consisting of ~ 8K annotations of complex events in ~ 4K stories. Each story is annotated from the perspective of a single participant. For the situation described in a story, an annotation is partially tailored to the identified participant taking on an "Agent-like" and "Patient-like" semantic role. Each annotation consists of several parts -- a high level process summary of the situation, its outcome and the changes resulting from it. More details on the annotations are presented below.

Here we release the collected annotations and the training, validation and test splits of the dataset. We also release the source code used for training and testing the BART and T5 models for which we reported results in the paper. Our paper "POQue: Asking Participant-specific Outcome questions for a Deeper Understanding of Complex Events" is published in the Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (https://aclanthology.org/2022.emnlp-main.pdf) and a direct link to the paper can be found here (https://aclanthology.org/2022.emnlp-main.594/).

The models -- BART and T5 -- presented in the paper were trained on the combined Train and Validation data splits and tested on the test split.

For this data collection, we took a holistic view of a complex event annotating various aspects of the complex event and its outcome as changes in participants. We show that while achieving a computational understanding of complex events is not simple, it can be made possible through semantically grounding the outcome of the complex event in the various changes of state in participants.

We selected stories from 3 English language datasets:

1. ROCStories (Mostafazadeh et al, 2016, https://www.aclweb.org/anthology/N16-1098/).

2. Stories used in the ESTER dataset (Han et al, 2021, https://aclanthology.org/2021.emnlp-main.597.pdf).

3. Heuristically salient portions of Annotated New York Times newswire articles (Sandhaus, 2008, https://catalog.ldc.upenn.edu/LDC2008T19)

We resolved coreferent mentions and selected the largest cluster of mentions. The shortest mention containing a name from the SSA names collection is identified as the Participant who is assigned either an "agent-like" or "patient-like" semantic role.

Annotators were presented a story and asked to fill out 4 steps in a HIT, where they identified the following aspects of the described situation:

1. A high level process summary. 2a) An endpoint for the situation. 2b) Whether the endpoint described is stated or implied by the story (A third option of unsure was also provided). 3a) Whether the identified Participant caused or experienced the situation. 3b) A summary of changes resulting from the situation. 3c) Various change modes undergone by the story partipants as a result of the situation.
2. Factors contributing to the endpoint and changes resulting from it.

1545 Random annotations were evaluated by 3 crowd workers and 2 experts. Each step of the 4 HITs were evaluated seperately without any cascading errors resulting in inter-annotator agreement scores of 0.74-0.96 using the weighted Fleiss's Kappa.

The data folder contains the following data files: all_train_data.json, all_valid_data.json, all_test_data.json

Each annotation is assigned a unique id which is a number in the range from 0 to 8219 (the total annotations collected with our AMT HIT). We discarded 447 annotations from 26 workers who did not follow our HIT instructions, and the remaining 7772 are distributed in the above 3 files. The fields collected for each annotation are as follows:

The src_code directory contains the following files used to train and evaluate models models on the 5 task formulations: model.py, task.py, test.py, train.py

This dataset was primarily built to advance research in understanding complex events. We formulated 5 tasks aimed at understanding various aspects of a complex event. We benchmarked BART and T5 models on these tasks using various automated and human evaluation metrics. Human evaluation consisted of 3 crowd workers assigning a 1-5 likert value for the evaluated attributes such as abstractness, factuality and salience. We report a few of the results here. Please refer to the paper for more results, our baselines and result analysis.

We fine-tuned base and large models to generate a high level abstract summary given either a story or factors. We evaluated this task on abstractness as determined by the average of human annotator likert scores and compared it with automated measurements of average length (the number of tokens in the endpoint description) and average extractiveness (the percentage of of trigrams in the summary that match with the story trigrams) for both reference and generated summaries. Please refer to the paper for automated evaluation scores such as Rouge, BLEU, etc.

We fine-tuned base and large models to generate the description of an endpoint given either a story or factors. We evaluated this task on factuality and salience as determined by the average of human annotator likert scores and compared these with automated measurements of average length (the number of tokens in the endpoint description) and average extractiveness (the percentage of of trigrams in the summary that match with the story trigrams) for both generated and reference factors. Please refer to the paper for automated evaluation scores such as Rouge, BLEU, etc.

We finetuned base and large models to generate the various factors that explain the complex event given the story and endpoint. We evaluated this task on brevity, factuality and salience as determined by the average of human annotator likert scores and compared these with automated measurements of average number of factors and average number of tokens across all factors for both generated and reference factors. Please refer to the paper for automated evaluation scores such as Rouge, BLEU, etc.

We finetuned base and large models to generate the various factors that explain the complex event given the story and endpoint. We evaluated this task on factuality and salience as determined by the average of human annotator likert scores for both the reference and generated summaries. Please refer to the paper for automated evaluation scores such as Rouge, BLEU, etc.

We finetuned models to classify the various change modes experienced by participants given the story and the changes resulting from the endpoint. We tested the T5 model with a classification head on top of the Encoder and in a text-to-text Encoder-Decoder setting. We evaluated this multi-label task on Subset Accuracy, Hamming Score and macro F1.

We finetuned base and large models to identify whether a participant was involved in causing or experiencing the event, given the story and the changes resulting from the endpoint. We also compared models trained seperately on agent and patient versions of the dataset. We evaluated this task using Accuracy and macro F1.

Please use the following BibTex entry to cite our work. We will update this when the formal proceedings are published.

@inproceedings{vallurupalli-etal-2022-poque,
    title = "{POQ}ue: Asking Participant-specific Outcome Questions for a Deeper Understanding of Complex Events",
    author = "Vallurupalli, Sai  and
      Ghosh, Sayontan  and
      Erk, Katrin  and
      Balasubramanian, Niranjan and
      Ferraro, Frank",
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = Dec,
    year = "2022",
    address = "Online and Abu Dhabi, UAE",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.emnlp-main.594/",
    doi = "",
    pages = "",
}

[Sai Vallurupalli], [Sayontan Ghosh], Katrin Erk, Niranjan Balasubramanian, Frank Ferraro