Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation

Liang, Weixin; Zou, James; Yu, Zhou

doi:10.48550/arxiv.2005.10716

Cited by 5 publications

(5 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, this plot may more reflect the UX of our response generators than a user preference for longer responses. These results may also reflect the inherent noise in user Likert-scale ratings (Liang et al, 2020).…”

Section: Analysis 61 Relationship Between Rating and Engagementmentioning

confidence: 97%

Neural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative Conversations

Paranjape,

See,

Kenealy

et al. 2020

Preprint

View full text Add to dashboard Cite

We present Chirpy Cardinal, an open-domain dialogue agent, as a research platform for the 2019 Alexa Prize competition. Building an open-domain socialbot that talks to real people is challenging -such a system must meet multiple user expectations such as broad world knowledge, conversational style, and emotional connection. Our socialbot engages users on their terms -prioritizing their interests, feelings and autonomy. As a result, our socialbot provides a responsive, personalized user experience, capable of talking knowledgeably about a wide variety of topics, as well as chatting empathetically about ordinary life. Neural generation plays a key role in achieving these goals, providing the backbone for our conversational and emotional tone. At the end of the competition, Chirpy Cardinal progressed to the finals with an average rating of 3.6/5.0, a median conversation duration of 2 minutes 16 seconds, and a 90 th percentile duration of over 12 minutes. * equal contribution 3rd Proceedings of Alexa Prize (Alexa Prize 2019).

show abstract

Section: Analysis 61 Relationship Between Rating and Engagementmentioning

confidence: 97%

Neural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative Conversations

Paranjape,

See,

Kenealy

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Pairwise versus single-model ratings Conversations are often either rated individually, e.g. with Likert-score ratings (Ashwin et al, 2017;Venkatesh et al, 2018;Zhang et al, 2018;Rashkin et al, 2019;See et al, 2019a;Dinan et al, 2019b, or pairwise by comparing models Liang et al, 2020;Vinyals and Le, 2015;Li et al, 2016;Lee et al, 2020). Likert scoring relies on absolute identification rather than relative discrimination, which is less reliable in humans (Stewart et al, 2005), leading to different biases per annotator (Kulikov et al, 2019).…”

Section: Existing Workmentioning

confidence: 99%

Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents

Smith¹,

Hsu²,

Qian³

et al. 2022

Preprint

View full text Add to dashboard Cite

At the heart of improving conversational AI is the open problem of how to evaluate conversations. Issues with automatic metrics are well known (Liu et al., 2016), with human evaluations still considered the gold standard. Unfortunately, how to perform human evaluations is also an open problem: differing data collection methods have varying levels of human agreement and statistical sensitivity, resulting in differing amounts of human annotation hours and labor costs. In this work we compare five different crowdworker-based human evaluation methods and find that different methods are best depending on the types of models compared, with no clear winner across the board. While this highlights the open problems in the area, our analysis leads to advice of when to use which one, and possible future directions.

show abstract

“…Other examples, like Elliot et al [7] and Mueser et al [25] found high correlations between rankings resulting from the evaluation of physical features in humans. Liang et al [20] proposes a model to 'calibrate' self-reported user ratings for dialogue systems due to issues with validity and bias. In relation to biomedical image assessments, where evaluation considers the visual quality of the stimuli, Phelps et al [27] found that pairwise comparisons and ranked Likert scores made for more accurate assessments in comparison to the use of non-ranked Likert scores.…”

Section: Related Workmentioning

confidence: 99%

To Rate or Not To Rate: Investigating Evaluation Methods for Generated Co-Speech Gestures

Wolfert

Girard

Kucherenko

et al. 2021

Proceedings of the 2021 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

While automatic performance metrics are crucial for machine learning of artificial human-like behaviour, the gold standard for evaluation remains human judgement. The subjective evaluation of artificial human-like behaviour in embodied conversational agents is however expensive and little is known about the quality of the data it returns. Two approaches to subjective evaluation can be largely distinguished, one relying on ratings, the other on pairwise comparisons. In this study we use co-speech gestures to compare the two against each other and answer questions about their appropriateness for evaluation of artificial behaviour. We consider their ability to rate quality, but also aspects pertaining to the effort of use and the time required to collect subjective data. We use crowd sourcing to rate the quality of co-speech gestures in avatars, assessing which method picks up more detail in subjective assessments. We compared gestures generated by three different machine learning models with various level of behavioural quality. We found that both approaches were able to rank the videos according to quality and that the ranking significantly correlated, showing that in terms of quality there is no preference of one method over the other. We also found that pairwise comparisons were slightly faster and came with improved inter-rater reliability, suggesting that for small-scale studies pairwise comparisons are to be favoured over ratings. CCS CONCEPTS• Human-centered computing → HCI design and evaluation methods; Human computer interaction (HCI).

show abstract

Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation

Cited by 5 publications

References 43 publications

Neural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative Conversations

Neural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative Conversations

Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents

To Rate or Not To Rate: Investigating Evaluation Methods for Generated Co-Speech Gestures

Contact Info

Product

Resources

About