Disentangling the Properties of Human Evaluation Methods: A Classification System to Support Comparability, Meta-Evaluation and Reproducibility Testing

Abstract

Current standards for designing and reporting human evaluations in NLP mean it is generally unclear which evaluations are comparable and can be expected to yield similar results when applied to the same system outputs. This has serious implications for reproducibility testing and meta-evaluation, in particular given that human evaluation is considered the gold standard against which the trustworthiness of automatic metrics is gauged. Using examples from NLG, we propose a classification system for evaluations based on disentangling (i) what is being evaluated (which aspect of quality), and (ii) how it is evaluated in specific (a) evaluation modes and (b) experimental designs. We show that this approach provides a basis for determining comparability, hence for comparison of evaluations across papers, meta-evaluation experiments, reproducibility testing.

A Reproduction Study of an Annotation-based Human Evaluation of MT Outputs

“…When evaluating Comprehensbility of a translated text, the source language text is not shown to evaluators. In terms of the classification system proposed by Belz et al (2020), Comprehensibility captures the goodness of both the form and content of a text in its own right, and is assessed here by a subjective, absolute, intrinsic evaluation measure.…”

Proceedings of the 15th International Conference on Natural Language Generation

“…There has been much discussion lately in the field of Natural Language Generation (NLG) focusing the need for evaluation benchmarks and standards, as evidenced by the prolific literature focusing on the issues surrounding human evaluation (Howcroft et al, 2020;Clark et al, 2021;Hämäläinen and Alnajjar, 2021;, as well as recently proposed benchmarks Khashabi et al, 2021;Mille et al, 2021). These are important and necessary debates -however, work has focused mainly on two-party dialogue systems.…”