Revisiting text decomposition methods for NLI-based factuality scoring of summaries

Glover, John; Fancellu, Federico; Jagannathan, V. K.; Gormley, Matthew R.; Schaaf, Thomas

doi:10.18653/v1/2022.gem-1.7

Cited by 3 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We denote those scores as factual-consistent, non-verified, and factual-inconsistent, respectively. This strategy seeks to address the shortcomings of traditional factuality metrics (Wang et al, 2020;Honovich et al, 2021;Glover et al, 2022a;Lee et al, 2023) that mainly depend on consistency with human-annotated references. These metrics often fail in emerging knowledge generation scenarios (Table 10), as they struggle with modelgenerated content beyond reference knowledge scope and face difficulties when references are unavailable in real-world applications.…”

Section: Intrinsic Evaluationmentioning

confidence: 99%

“…We chose the factCC version as our baseline. NLI-decompose-claim (Glover et al, 2022b) found that in general, sentence-level decomposition is preferable for the hypothesis side of the NLI input. So we also decompose the generated knowledge into sentences and then aggregate the sentence-level scores to produce a document-level score.…”

Section: Datasetmentioning

confidence: 99%

“…However, their time-consuming nature and subjectivity often encounter issues of scalability and reproducibility. Secondly, datasets have been constructed to evaluate open-domain generation with the aid of references (Honovich et al, 2021;Glover et al, 2022a;Lee et al, 2023;Li et al, 2023). These methods, while more objective, are limited by their dependence on humanlabelled references, impacting their real-world applicability and generalizability to dynamically generated content.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators

Chen,

Deng,

Bian

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Large language models (LLMs) outperform information retrieval techniques for downstream knowledge-intensive tasks when being prompted to generate world knowledge. Yet, community concerns abound regarding the factuality and potential implications of using this uncensored knowledge. In light of this, we introduce CONNER, a COmpreheNsive kNowledge Evaluation fRamework, designed to systematically and automatically evaluate generated knowledge from six important perspectives -Factuality, Relevance, Coherence, Informativeness, Helpfulness and Validity. We conduct an extensive empirical analysis of the generated knowledge from three different types of LLMs on two widely-studied knowledge-intensive tasks, i.e., open-domain question answering and knowledge-grounded dialogue. Surprisingly, our study reveals that the factuality of generated knowledge, even if lower, does not significantly hinder downstream tasks. Instead, the relevance and coherence of the outputs are more important than small factual mistakes. Further, we show how to use CONNER to improve knowledge-intensive tasks by designing two strategies: Prompt Engineering and Knowledge Selection. Our evaluation code and LLMgenerated knowledge with human annotations will be released 1 to facilitate future research.

show abstract

Section: Intrinsic Evaluationmentioning

confidence: 99%

Section: Datasetmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation