SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization

Vig, Jesse; Kryściński, Wojciech; Goel, Karan; Rajani, Nazneen Fatema

doi:10.18653/v1/2021.acl-demo.18

Cited by 5 publications

(5 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…If a model can pinpoint which portion of a summary is inconsistent, some work has shown that corrector models can effectively re-write the problematic portions and often remove the inconsistency . Furthermore, fine-grained consistency scores can be incorporated into visual analysis tools for summarization such as Summ-Viz (Vig et al, 2021). The SUMMAC ZS model is directly interpretable, whereas the SUMMAC CONV is slightly more opaque, due to the inability to trace back a low score to a single sentence in the document being invalidated.…”

Section: Choice Of Granularitymentioning

confidence: 99%

SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization

Laban

Schnabel

Bennett

et al. 2022

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

In the summarization domain, a key requirement for summaries is to be factually consistent with the input document. Previous work has found that natural language inference (NLI) models do not perform competitively when applied to inconsistency detection. In this work, we revisit the use of NLI for inconsistency detection, finding that past work suffered from a mismatch in input granularity between NLI datasets (sentence-level), and inconsistency detection (document level). We provide a highly effective and light-weight method called SummaCConv that enables NLI models to be successfully used for this task by segmenting documents into sentence units and aggregating scores between pairs of sentences. We furthermore introduce a new benchmark called SummaC (Summary Consistency) which consists of six large inconsistency detection datasets. On this dataset, SummaCConv obtains state-of-the-art results with a balanced accuracy of 74.4%, a 5% improvement compared with prior work.

show abstract

Section: Choice Of Granularitymentioning

confidence: 99%

SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization

Laban

Schnabel

Bennett

et al. 2022

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…However, there is a need to address issues such as a proper model and data analysis tool and understanding the failure model of summarization. SummVis [22], an open-source tool, allows us to visualize, generate a summary, and analyze the summarization models and the evaluation metrics used. Topic modeling has been recently used in text summarization to identify hidden topics in the document [23].…”

Section: Literature Reviewmentioning

confidence: 99%

A hybrid approach for text summarization using semantic latent Dirichlet allocation and sentence concept mapping with transformer

Gurusamy,

Rengarajan,

Srinivasan

2023

IJECE

View full text Add to dashboard Cite

<span lang="EN-US">Automatic text summarization generates a summary that contains sentences reflecting the essential and relevant information of the original documents. Extractive summarization requires semantic understanding, while abstractive summarization requires a better intermediate text representation. This paper proposes a hybrid approach for generating text summaries that combine extractive and abstractive methods. To improve the semantic understanding of the model, we propose two novel extractive methods: semantic latent Dirichlet allocation (semantic LDA) and sentence concept mapping. We then generate an intermediate summary by applying our proposed sentence ranking algorithm over the sentence concept mapping. This intermediate summary is input to a transformer-based abstractive model fine-tuned with a multi-head attention mechanism. Our experimental results demonstrate that the proposed hybrid model generates coherent summaries using the intermediate extractive summary covering semantics. As we increase the concepts and number of words in the summary the rouge scores are improved for precision and F1 scores in our proposed model.</span>

show abstract

“…In Section 2, we found several mentioned human evaluation is challenging since annotators need to read long source documents. Some prior work has suggested highlighting spans in the source document that align with the summary (Hardy et al, 2019;Kryscinski et al, 2020;Vig et al, 2021) as shown in Figure 1. However, these efforts have exclusively focused on news summarization with relatively short source documents, like CNN/DM (804 words) (Nallapati et al, 2016) or XSUM (438 words) (Narayan et al, 2018).…”

Section: S|mentioning

confidence: 99%

LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization

Krishna,

Bransom,

Kuehl

et al. 2023

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics

View full text Add to dashboard Cite

While human evaluation remains best practice for accurately judging the faithfulness of automatically-generated summaries, few solutions exist to address the increased difficulty and workload when evaluating long-form summaries. Through a survey of 162 papers on long-form summarization, we first shed light on current human evaluation practices surrounding long-form summaries. We find that 73% of these papers do not perform any human evaluation on model-generated summaries, while other works face new difficulties that manifest when dealing with long documents (e.g., low inter-annotator agreement). Motivated by our survey, we present LONGEVAL, a set of guidelines for human evaluation of faithfulness in long-form summaries that addresses the following challenges: (1) How can we achieve high inter-annotator agreement on faithfulness scores? (2) How can we minimize annotator workload while maintaining accurate faithfulness scores? and (3) Do humans benefit from automated alignment between summary and source snippets? We deploy LONGEVAL in annotation studies on two long-form summarization datasets in different domains (SQuALITY and PubMed), and we find that switching to a finer granularity of judgment (e.g., clause-level) reduces inter-annotator variance in faithfulness scores (e.g., std-dev from 18.5 to 6.8). We also show that scores from a partial annotation of fine-grained units highly correlates with scores from a full annotation workload (0.89 Kendall's τ using 50% judgments). We release our human judgments, annotation templates, and our software for future research.

show abstract

SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization

Cited by 5 publications

References 20 publications

SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization

SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization

A hybrid approach for text summarization using semantic latent Dirichlet allocation and sentence concept mapping with transformer

LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization

Contact Info

Product

Resources

About