Assessing The Factual Accuracy of Generated Text

Goodrich, Ben; Rao, Vinay; Saleh, M. A.; Liu, Peter Jianrui

doi:10.1145/3292500.3330955

Cited by 137 publications

(122 citation statements)

References 27 publications

Supporting

Mentioning

120

Contrasting

Unclassified

Order By: Relevance

“…Recent work in addressing faithfulness of text generations can be broadly divided into three groups: structured information based, multi-task formulations, and post-processing methods. The first group leverages structured knowledge, like Open IE triples (Cao et al, 2018;Goodrich et al, 2019), dependency trees (Song et al, 2018), or generated semantic roles (Fan et al, 2018) as additional input for generation. However, incorporation of these as additional embeddings in model architectures does not explain how these influence model generations.…”

Section: Related Workmentioning

confidence: 99%

Evaluating Factuality in Generation with Dependency-level Entailment

Goyal

Durrett

2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Despite significant progress in text generation models, a serious limitation is their tendency to produce text that is factually inconsistent with information in the input. Recent work has studied whether textual entailment systems can be used to identify factual errors; however, these sentence-level entailment models are trained to solve a different problem than generation filtering and they do not localize which part of a generation is non-factual. In this paper, we propose a new formulation of entailment that decomposes it at the level of dependency arcs. Rather than focusing on aggregate decisions, we instead ask whether the semantic relationship manifested by individual dependency arcs in the generated output is supported by the input. Human judgments on this task are difficult to obtain; we therefore propose a method to automatically create data based on existing entailment or paraphrase corpora. Experiments show that our dependency arc entailment model trained on this data can identify factual inconsistencies in paraphrasing and summarization better than sentencelevel methods or those based on question generation, while additionally localizing the erroneous parts of the generation. 1

show abstract

Section: Related Workmentioning

confidence: 99%

Evaluating Factuality in Generation with Dependency-level Entailment

Goyal

Durrett

2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

show abstract

“…Then they train a fact-checking model to classify the label of the claim and extract spans in both the source document and the generated summary explaining the model's decision. Goodrich et al (2019) introduced a model-based…”

Section: Related Workmentioning

confidence: 99%

Reducing Quantity Hallucinations in Abstractive Summarization

Zheng¹,

Cohen²,

Webber³

2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

It is well-known that abstractive summaries are subject to hallucination-including material that is not supported by the original text. While summaries can be made hallucinationfree by limiting them to general phrases, such summaries would fail to be very informative. Alternatively, one can try to avoid hallucinations by verifying that any specific entities in the summary appear in the original text in a similar context. This is the approach taken by our system, HERMAN. The system learns to recognize and verify quantity entities (dates, numbers, sums of money, etc.) in a beamworth of abstractive summaries produced by state-of-the-art models, in order to up-rank those summaries whose quantity terms are supported by the original text. Experimental results demonstrate that the ROUGE scores of such up-ranked summaries have a higher Precision than summaries that have not been upranked, without a comparable loss in Recall, resulting in higher F 1 . Preliminary human evaluation of up-ranked vs. original summaries shows people's preference for the former.

show abstract

“…Additionally, Goodrich et al (2019) compare several models such as relation extraction, binary classification and end-to-end models (E2E) for estimating factual accuracy on a Wikipedia text summarization task. They show that their E2E model for factual correctness has the highest correlation with human judgements and suggest that the E2E models could benefit from a better labeling scheme.…”

Section: Factual Errors In Summariesmentioning

confidence: 99%

“…We find that previous research has not established a detailed typology of summarization errors. Most work instead relies on on a binary distinction between correct and erroneous (Cao et al, 2018;Falke et al, 2019;Lebanoff et al, 2019) or faithfulness measured on a Likert scale (Goodrich et al, 2019). However, not all errors are created equal.…”

Section: Factual Errors In Summariesmentioning

confidence: 99%

Truth or Error? Towards systematic analysis of factual errors in abstractive summaries

Lux¹,

Sappelli²,

Larson³

2020

Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems

View full text Add to dashboard Cite

This paper presents a typology of errors produced by automatic summarization systems. The typology was created by manually analyzing the output of four recent neural summarization systems. Our work is motivated by the growing awareness of the need for better summary evaluation methods that go beyond conventional overlap-based metrics. Our typology is structured into two dimensions. First, the Mapping Dimension describes surface-level errors and provides insight into word-sequence transformation issues. Second, the Meaning Dimension describes issues related to interpretation and provides insight into breakdowns in truth, i.e., factual faithfulness to the original text. Comparative analysis revealed that two neural summarization systems leveraging pretrained models have an advantage in decreasing grammaticality errors, but not necessarily factual errors. We also discuss the importance of ensuring that summary length and abstractiveness do not interfere with evaluating summary quality.

show abstract

Assessing The Factual Accuracy of Generated Text

Cited by 137 publications

References 27 publications

Evaluating Factuality in Generation with Dependency-level Entailment

Evaluating Factuality in Generation with Dependency-level Entailment

Reducing Quantity Hallucinations in Abstractive Summarization

Truth or Error? Towards systematic analysis of factual errors in abstractive summaries

Contact Info

Product

Resources

About