SummVis is an interactive visualization tool for abstractive summarization systems, supporting analysis of models, data, and evaluation metrics.
Authors: Jesse Vig1,
Wojciech Kryściński1,
Karan Goel2,
Nazneen Fatema Rajani1
1Salesforce Research 2Stanford Hazy Research
Note: SummVis is under active development, so expect continued updates in the coming weeks and months. Feel free to raise issues for questions, suggestions, requests or bug reports.
The SummVis interface is shown below. The primary components are: (a) configuration panel, (b) source document (or reference summary, depending on configuration), (c) generated summaries (and/or reference summary, depending on configuration), (d) scroll bar with global view of annotations.
Solid underlines align n-grams between source document and the selected summary (BART). Novel words in the summary that do not appear in the source document are bolded, while novel entities are bolded in red. Stopwords are grayed out and are not used in the matching algorithms. Dotted underlines indicate tokens that are semantically related to a token in the source document. You may hover over a token to see the most semantically similar tokens in the source document (see inset image), or click on the token to auto-scroll the source document to the most similar token.
IMPORTANT: Please use python>=3.8
since some dependencies require that for installation.
git clone https://github.com/robustness-gym/summvis.git
cd summvis
pip install -r requirements.txt
python -m spacy download en_core_web_sm
Installation takes around 2 minutes on a Macbook Pro.
Follow the steps below to start using SummVis immediately.
Download our pre-cached dataset that contains predictions for state-of-the-art models such as PEGASUS and BART on 1000 examples taken from the CNN / Daily Mail validation set.
mkdir data
mkdir preprocessing
curl https://storage.googleapis.com/sfr-summvis-data-research/cnn_dailymail_1000.validation.anonymized.zip --output preprocessing/cnn_dailymail_1000.validation.anonymized.zip
unzip preprocessing/cnn_dailymail_1000.validation.anonymized.zip -d preprocessing/
Next, we'll need to add the original examples from the CNN / Daily Mail dataset to deanonymize the data (this information
is omitted for copyright reasons). The preprocessing.py
script can be used for this with the --deanonymize
flag.
python preprocessing.py \
--deanonymize \
--dataset_rg preprocessing/cnn_dailymail_1000.validation.anonymized \
--dataset cnn_dailymail \
--version 3.0.0 \
--split validation \
--processed_dataset_path data/try:cnn_dailymail_1000.validation \
--try_it
This will take either a few seconds or a few minutes depending on whether you've previously loaded CNN/DailyMail from the Datasets library.
Finally, we're ready to run the Streamlit app. Once the app loads, make sure it's pointing to the right File
at the top
of the interface.
streamlit run summvis.py
CNN / Daily Mail (1000 examples from validation set): https://storage.googleapis.com/sfr-summvis-data-research/cnn_dailymail_1000.validation.anonymized.zip
CNN / Daily Mail (full validation set): https://storage.googleapis.com/sfr-summvis-data-research/cnn_dailymail.validation.anonymized.zip
XSum (1000 examples from validation set): https://storage.googleapis.com/sfr-summvis-data-research/xsum_1000.validation.anonymized.zip
XSum (full validation set): https://storage.googleapis.com/sfr-summvis-data-research/xsum.validation.anonymized.zip
We recommend that you choose the smallest dataset that fits your need in order to minimize download / preprocessing time.
mkdir data
mkdir preprocessing
curl https://storage.googleapis.com/sfr-summvis-data-research/cnn_dailymail_1000.validation.anonymized.zip --output preprocessing/cnn_dailymail_1000.validation.anonymized.zip
unzip preprocessing/cnn_dailymail_1000.validation.anonymized.zip -d preprocessing/
Set the --n_samples
argument and name the --processed_dataset_path
output file accordingly.
python preprocessing.py \
--deanonymize \
--dataset_rg preprocessing/cnn_dailymail_1000.validation.anonymized \
--dataset cnn_dailymail \
--version 3.0.0 \
--split validation \
--processed_dataset_path data/100:cnn_dailymail_1000.validation \
--n_samples 100
python preprocessing.py \
--deanonymize \
--dataset_rg preprocessing/cnn_dailymail_1000.validation.anonymized \
--dataset cnn_dailymail \
--version 3.0.0 \
--split validation \
--processed_dataset_path data/full:cnn_dailymail_1000.validation \
--n_samples 1000
python preprocessing.py \
--deanonymize \
--dataset_rg preprocessing/cnn_dailymail.validation.anonymized \
--dataset cnn_dailymail \
--version 3.0.0 \
--split validation \
--processed_dataset_path data/full:cnn_dailymail.validation
python preprocessing.py \
--deanonymize \
--dataset_rg preprocessing/xsum_1000.validation.anonymized \
--dataset xsum \
--split validation \
--processed_dataset_path data/full:xsum_1000.validation \
--n_samples 1000
Once the app loads, make sure it's pointing to the right File
at the top
of the interface.
streamlit run summvis.py
Alternately, if you need to point SummVis to a folder where your data is stored.
streamlit run summvis.py -- --path your/path/to/data
Note that the additional --
is not a mistake, and is required to pass command-line arguments in streamlit.
You can also perform preprocessing end-to-end to load any summarization dataset or model predictions into SummVis. Instructions for this are provided below.
Prior to running the following, an additional install step is required:
python -m spacy download en_core_web_lg
Loads in a dataset from HF, or any dataset that you have and stores it in a
standardized format with columns for document
and summary:reference
.
python preprocessing.py \
--standardize \
--dataset cnn_dailymail \
--version 3.0.0 \
--split validation \
--save_jsonl_path preprocessing/cnn_dailymail.validation.jsonl
python preprocessing.py \
--standardize \
--dataset_jsonl path/to/my_dataset.jsonl \
--doc_column name_of_document_column \
--reference_column name_of_reference_summary_column \
--save_jsonl_path preprocessing/my_dataset.jsonl
Takes a saved dataset that has already been standardized and adds predictions to it from prediction jsonl files. Cached predictions for several models available here: https://storage.googleapis.com/sfr-summvis-data-research/predictions.zip
You may also generate your own predictions using this this script.
python preprocessing.py \
--join_predictions \
--dataset_jsonl preprocessing/cnn_dailymail.validation.jsonl \
--prediction_jsonls \
predictions/bart-cnndm.cnndm.validation.results.anonymized \
predictions/bart-xsum.cnndm.validation.results.anonymized \
predictions/pegasus-cnndm.cnndm.validation.results.anonymized \
predictions/pegasus-multinews.cnndm.validation.results.anonymized \
predictions/pegasus-newsroom.cnndm.validation.results.anonymized \
predictions/pegasus-xsum.cnndm.validation.results.anonymized \
--save_jsonl_path preprocessing/cnn_dailymail.validation.jsonl
Takes a saved dataset that has been standardized, and predictions already added.
Applies all the preprocessing steps to it (running spaCy
, lexical and semantic aligners),
and stores the processed dataset back to disk.
python preprocessing.py \
--workflow \
--dataset_jsonl preprocessing/cnn_dailymail.validation.jsonl \
--processed_dataset_path data/cnn_dailymail.validation \
--try_it
python preprocessing.py \
--workflow \
--dataset_jsonl preprocessing/cnn_dailymail.validation.jsonl \
--processed_dataset_path data/cnn_dailymail
When referencing this repository, please cite this paper:
@misc{vig2021summvis,
title={SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization},
author={Jesse Vig and Wojciech Kryscinski and Karan Goel and Nazneen Fatema Rajani},
year={2021},
eprint={2104.07605},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2104.07605}
}
We thank Michael Correll for his valuable feedback.