How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluation

Abstract

Manual evaluation is essential to judge progress on automatic text summarization. However, we conduct a survey on recent summarization system papers that reveals little agreement on how to perform such evaluation studies. We conduct two evaluation experiments on two aspects of summaries' linguistic quality (coherence and repetitiveness) to compare Likert-type and ranking annotations and show that best choice of evaluation method can vary from one aspect to another. In our survey, we also find that study parameters such as the overall number of annotators and distribution of annotators to annotation items are often not fully reported and that subsequent statistical analysis ignores grouping factors arising from one annotator judging multiple summaries. Using our evaluation experiments, we show that the total number of annotators can have a strong impact on study power and that current statistical analysis methods can inflate type I error rates up to eight-fold. In addition, we highlight that for the purpose of system comparison the current practice of eliciting multiple judgements per summary leads to less powerful and reliable annotations given a fixed study budget.

Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries

“…We follow Steen and Markert (2021) and report Krippendorff's alpha and Split-Half Reliability as measures of the reliability of crowdsourced annotations. Krippendorff's alpha (α) is a reliability coefficient developed to measure the agreement among multiple annotators (Krippendorff, 2011).…”

“…We follow a similar block-design described in Steen and Markert (2021). We note that we include the input document as the context of the summaries as opposed to the coherence and repetition dimensions studied in that work, which do not require reading the input article.…”

“…We include all 4 generated summaries for each document in the same block, resulting in 5 × 4 = 20 summaries per block. We require 3 annotators per block as in Steen and Markert (2021), and each annotator is limited to annotating at most two blocks total across all tasks. A further study of the effect of the number of annotators or block design is left for future work.…”

“…The Likert Scale (LS) is a common rating-based evaluation protocol (Asghar et al, 2018). Likert Scales applied to summarization typically range from 1-5 (Steen and Markert, 2021). Best-Worst Scaling (BWS) is a type of ranking-oriented evaluation that requires annotators to specify only the best and the worst example in a set of summaries (Hollis and Westbury, 2018;Kiritchenko and Mohammad, 2017).…”

“…Prior research has claimed that Best-Worst Scaling produces higher-quality evaluations than rating scales such as the Likert Scale for tasks such as sentiment analysis (Kiritchenko and Mohammad, 2017). In the context of summarization, Steen and Markert (2021) find that, compared to the Likert Scale, ranking-based protocols are more reliable for measuring summary coherence but less so for repetition. However, previous studies have not analyzed annotation reliability in the context of factual consistency for summarization.…”