SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization

Vig, Jesse; Kryściński, Wojciech; Goel, Karan; Rajani, Nazneen Fatema

doi:10.48550/arxiv.2104.07605

Cited by 3 publications

(4 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…If a model can pinpoint which portion of a summary is inconsistent, some work has shown that corrector models can effectively re-write the problematic portions and often remove the inconsistency . Furthermore, fine-grained consistency scores can be incorporated into visual analysis tools for summarization such as SummViz (Vig et al, 2021). The SUMMAC ZS model is directly interpretable, whereas the SUMMAC Conv is slightly more opaque, due to the inability to trace back a low score to a single sentence in the document being invalidated.…”

Section: Choice Of Granularitymentioning

confidence: 99%

SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization

Laban¹,

Schnabel²,

Bennett³

et al. 2021

Preprint

View full text Add to dashboard Cite

In the summarization domain, a key requirement for summaries is to be factually consistent with the input document. Previous work has found that natural language inference (NLI) models do not perform competitively when applied to inconsistency detection. In this work, we revisit the use of NLI for inconsistency detection, finding that past work suffered from a mismatch in input granularity between NLI datasets (sentence-level), and inconsistency detection (document level). We provide a highly effective and light-weight method called SUMMAC Conv that enables NLI models to be successfully used for this task by segmenting documents into sentence units and aggregating scores between pairs of sentences. On our newly introduced benchmark called SUMMAC (Summary Consistency) consisting of six large inconsistency detection datasets, SUMMAC Conv obtains stateof-the-art results with a balanced accuracy of 74.4%, a 5% point improvement compared to prior work.

show abstract

Section: Choice Of Granularitymentioning

confidence: 99%

SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization

Laban¹,

Schnabel²,

Bennett³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Sum-merTime adopts a subset of such metrics in Sum-mEval that are more popular and easier to understand. SummerTime also works well with Sum-mVis (Vig et al, 2021), which provides an interactive way of analysing summarization results on the token-level. We also allow SummerTime to store output in a format that can be directly used by SummVis and its UI.…”

Section: Existing Systems For Summarizationmentioning

confidence: 99%

“…Visualization In addition to showing the numerical results as tables, SummerTime also allows the users to visualize the differences between different models with different charts and SummVis (Vig et al, 2021). Fig.…”

Section: Model Selectionmentioning

confidence: 99%

SummerTime: Text Summarization Toolkit for Non-experts

Ni¹,

Zhangir²,

Mutuma³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent advances in summarization provide models that can generate summaries of higher quality. Such models now exist for a number of summarization tasks, including query-based summarization, dialogue summarization, and multi-document summarization. While such models and tasks are rapidly growing in the research field, it has also become challenging for non-experts to keep track of them. To make summarization methods more accessible to a wider audience, we develop SummerTime by rethinking the summarization task from the perspective of an NLP non-expert. Summer-Time is a complete toolkit for text summarization, including various models, datasets and evaluation metrics, for a full spectrum of summarization-related tasks. SummerTime integrates with libraries designed for NLP researchers, and enables users with easy-touse APIs. With SummerTime, users can locate pipeline solutions and search for the best model with their own data, and visualize the differences, all with a few lines of code. We also provide explanations for models and evaluation metrics to help users understand the model behaviors and select models that best suit their needs. Our library, along with a notebook demo, is available at https://github.com/Yale-LILY/ SummerTime.

show abstract

“…Yet other methods exist for domains beyond supervised learning / classification, applied instead to unsupervised learning / clustering [19,32,68], reinforcement learning [53], AI planning [15], computer vision [91], recommendation systems [88], natural language processing [57,81,86], speech recognition [33], or multi-agent simulations [6] . Specialized XAI techniques even exist for adaptive systems such as interactive visualization systems [77], interactive virtual agents [37,83], active learning [27], and human-in-the-loop systems [86] . Furthermore, comparative studies across multiple XAI techniques have shown low mutual agreement [58], which is consistent with internal research findings at AI Research.…”

Section: The Challengesmentioning

confidence: 99%

Seven challenges for harmonizing explainability requirements

Chen,

Storchan

2021

Preprint

View full text Add to dashboard Cite

Regulators have signalled an interest in adopting explainable AI (XAI) techniques to handle the diverse needs for model governance, operational servicing, and compliance in the financial services industry. In this short overview, we review the recent technical literature in XAI and argue that based on our current understanding of the field, the use of XAI techniques in practice necessitate a highly contextualized approach considering the specific needs of stakeholders for particular business applications.

show abstract

SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization

Cited by 3 publications

References 16 publications

SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization

SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization

SummerTime: Text Summarization Toolkit for Non-experts

Seven challenges for harmonizing explainability requirements

Contact Info

Product

Resources

About