Best practices for the human evaluation of automatically generated text

Lee, Chris van der; Gatt, Albert; Miltenburg, Emiel van; Wubben, Sander; Krahmer, Emiel

doi:10.18653/v1/w19-8643

Cited by 125 publications

(102 citation statements)

References 82 publications

(95 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Because of the minimal difference of 0.04, we decided to still use the PU measure in our further analysis for interpretation. With these results, we achieved a better agreement level than the average expert agreement of summarization evaluation reported in other papers Van Der Lee et al (2019).…”

Section: Comparing Crowd With Expertsupporting

confidence: 51%

“…Therefore, more and more researchers refrain from using automatic metrics as a primary evaluation method (Reiter, 2018). Still, Van Der Lee et al (2019) report that 80% of the empirical papers presented at the ACL track on NLG or at the INLG conference in 2018 using automatic metrics due to the lack of alternatives and the fast and cost-effective nature.…”

Section: Untrained Automatic Metricsmentioning

confidence: 99%

“…Even though there has been an enormous increase in automatic summarization research, human evaluation of summarization is still an understudied aspect. One the one hand, there is no standard procedure for conducting human evaluation, which is leading to a high degree of variation and different results (Van Der Lee et al, 2019); on the other hand, human evaluation is usually carried out in a traditional laboratory environment by linguistic experts, which is costly and time-consuming to run and prone to subjective biases (Celikyilmaz et al, 2020). Therefore, automatic evaluation metrics such as BLEU and ROUGE have been used as substitutes for human evaluation (Papineni et al, 2002;Lin, 2004).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Best Practices for Crowd-based Evaluation of German Summarization: Comparing Crowd, Expert and Automatic Evaluation

Iskender¹,

Polzehl²,

Möller³

2020

Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems

View full text Add to dashboard Cite

One of the main challenges in the development of summarization tools is summarization quality evaluation. On the one hand, the human assessment of summarization quality conducted by linguistic experts is slow, expensive, and still not a standardized procedure. On the other hand, the automatic assessment metrics are reported not to correlate high enough with human quality ratings. As a solution, we propose crowdsourcing as a fast, scalable, and costeffective alternative to expert evaluations to assess the intrinsic and extrinsic quality of summarization by comparing crowd ratings with expert ratings and automatic metrics such as ROUGE, BLEU, or BertScore on a German summarization data set. Our results provide a basis for best practices for crowd-based summarization evaluation regarding major influential factors such as the best annotation aggregation method, the influence of readability and reading effort on summarization evaluation, and the optimal number of crowd workers to achieve comparable results to experts, especially when determining factors such as overall quality, grammaticality, referential clarity, focus, structure & coherence, summary usefulness, and summary informativeness.

show abstract

Section: Comparing Crowd With Expertsupporting

confidence: 51%

Section: Untrained Automatic Metricsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Best Practices for Crowd-based Evaluation of German Summarization: Comparing Crowd, Expert and Automatic Evaluation

Iskender¹,

Polzehl²,

Möller³

2020

Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems

View full text Add to dashboard Cite

show abstract

“…The choice for a 5-point scale lies in the fact that this type of rating system is the most used in human evaluation tasks and recommended in numerous studies. In essence, this scale gives more reliable results than the ones with finer granularity because it seems to be easier to understand and handle for subjects (Korshunov et al, 2015;Sinkowitz et al, 2013, van der Lee, 2019.…”

Section: Methodsmentioning

confidence: 99%

Deep Multitask Gaze Estimation with a Constrained Landmark-Gaze Model

Liu

Odobez

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The step preceding the speaker identification process consists in the determination of the authenticity of a speech sample. The focus of this thesis is on the performance of humans in detecting altered samples from replay, speech synthesis (TTS) and voice conversion (VC) systems. A listening test was constructed on the online survey platform LimeSurvey. The participants were asked to assess a series of recordings by first giving a binary evaluation ("authentic" or "altered") and then by specifying their level of confidence on a 5-point scale. Moreover, the logic behind the aural approach was studied by inspecting the criteria used by the respondents in the assessment of the recordings. The same samples were also evaluated with an automatic LFCC-GMM-based system, trained on two different datasets, in order to make a comparison. The results show that the human's performance (EER=0.10) surpasses the machine's (EER=0.35 and EER=0.46) in the detection of the altered samples. However, sophisticated voice disguise systems have now reached levels of quality that can also easily fool most humans, which makes them a real threat to security and the identification process.

show abstract

“…Human-Bot Conversations. In order to perform interactive multi-turn evaluations, the standard method is to let humans converse with a chatbot and rate it afterward (Ghandeharioun et al, 2019), typically using Likert scales (van der Lee et al, 2019). The ConvAI2 challenge (Dinan et al, 2020b) and the Alexa Prize (Venkatesh et al, 2018) applied this procedure.…”

Section: Related Workmentioning

confidence: 99%

Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems

Deriu

Tuggener

Däniken³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

The lack of time-efficient and reliable evaluation methods hamper the development of conversational dialogue systems (chatbots). Evaluations requiring humans to converse with chatbots are time and cost-intensive, put high cognitive demands on the human judges, and yield low-quality results. In this work, we introduce Spot The Bot, a cost-efficient and robust evaluation framework that replaces human-bot conversations with conversations between bots. Human judges then only annotate for each entity in a conversation whether they think it is human or not (assuming there are humans participants in these conversations). These annotations then allow us to rank chatbots regarding their ability to mimic the conversational behavior of humans. Since we expect that all bots are eventually recognized as such, we incorporate a metric that measures which chatbot can uphold human-like behavior the longest, i.e., Survival Analysis. This metric has the ability to correlate a bot's performance to certain of its characteristics (e.g., fluency or sensibleness), yielding interpretable results. The comparably low cost of our framework allows for frequent evaluations of chatbots during their evaluation cycle. We empirically validate our claims by applying Spot The Bot to three domains, evaluating several stateof-the-art chatbots, and drawing comparisons to related work. The framework is released as a ready-to-use tool.

show abstract

Best practices for the human evaluation of automatically generated text

Cited by 125 publications

References 82 publications

Best Practices for Crowd-based Evaluation of German Summarization: Comparing Crowd, Expert and Automatic Evaluation

Best Practices for Crowd-based Evaluation of German Summarization: Comparing Crowd, Expert and Automatic Evaluation

Deep Multitask Gaze Estimation with a Constrained Landmark-Gaze Model

Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems

Contact Info

Product

Resources

About