Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries

Abstract

Current pre-trained models applied for summarization are prone to factual inconsistencies that misrepresent the source text. Evaluating the factual consistency of summaries is thus necessary to develop better models. However, the human evaluation setup for evaluating factual consistency has not been standardized. To determine the factors that affect the reliability of the human evaluation, we crowdsource evaluations for factual consistency across stateof-the-art models on two news summarization datasets using the rating-based Likert Scale and ranking-based Best-Worst Scaling. Our analysis reveals that the ranking-based Best-Worst Scaling offers a more reliable measure of summary quality across datasets and that the reliability of Likert ratings highly depends on the target dataset and the evaluation design. To improve crowdsourcing reliability, we extend the scale of the Likert rating and present a scoring algorithm for Best-Worst Scaling that we call value learning. Our crowdsourcing guidelines will be publicly available to facilitate future work on factual consistency in summarization.

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text

“…Kiritchenko and Mohammad (2017) demonstrated that best-worst scaling (asking evaluators to choose the best and the worst items in a set) is an efficient and reliable method for collecting annotations, and this approach has been used to collect comparative evaluations of generated text (e.g., Liu & Lapata, 2019;Amplayo et al, 2021). Best-worst scaling was also more recently been shown as a more effective approach than Likert scales for assessing factual consistency of summaries (Tang et al, 2022). Belz and Kow (2011) further compare continuous and discrete rating scales and found that both lead to similar results, but raters preferred continuous scales, consistent with prior findings (Svensson, 2000).…”

Grounded and Transparent Response Generation for Conversational Information-Seeking Systems

“…Generative language models have been widely adopted for response generation [14,28]; however, in the realm of open-ended information-seeking dialogues, the assumption that a user's query can be definitively answered by simply summarizing information from top retrieved passages falls short of reality. System responses are susceptible to various limitations, such as the failure to find a response which may result in hallucinations [10], providing a biased response only partially answering the question [9], or even presenting content with factual errors [26]. Consequently, relying solely on summarizing relevant information may lead to providing users with biased, incomplete, or, worse, incorrect responses [26].…”

“…System responses are susceptible to various limitations, such as the failure to find a response which may result in hallucinations [10], providing a biased response only partially answering the question [9], or even presenting content with factual errors [26]. Consequently, relying solely on summarizing relevant information may lead to providing users with biased, incomplete, or, worse, incorrect responses [26].…”