Evaluating Dialogue Generation Systems via Response Selection

Sato, Shiki; Akama, Reina; Ouchi, Hiroki; Suzuki, Jun; Inui, Kentaro

doi:10.5715/jnlp.27.677

Cited by 1 publication

(1 citation statement)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, ADEM can't discriminate between gold responses and certain classes of adversarial negatives e.g reversed gold responses and repeating the context as the response (Sai et al, 2019). Sato et al (2020) evaluate dialog systems through their ability at selecting valid responses from a semi-automatically curated candidate list. Mehri and Eskenazi (2020b) introduce the unsupervised, reference-free USR metric, which leverages a suite of RoBERTa (Liu et al, 2019) models, each finetuned to score one of five dialog aspects e.g Natural and Uses Knowledge.…”

Section: Related Workmentioning

confidence: 99%

Improving Automated Evaluation of Open Domain Dialog via Diverse Reference Augmentation

Gangal¹,

Jhamtani²,

Hovy³

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

Multiple different responses are often plausible for a given open domain dialog context. Prior work has shown the importance of having multiple valid reference responses for meaningful and robust automated evaluations. In such cases, common practice has been to collect more human written references. However, such collection can be expensive, time consuming, and not easily scalable. Instead, we propose a novel technique for automatically expanding a human generated reference to a set of candidate references. We fetch plausible references from knowledge sources, and adapt them so that they are more fluent in context of the dialog instance in question. More specifically, we use (1) a commonsense knowledge base to elicit a large number of plausible reactions given the dialog history (2) relevant instances retrieved from dialog corpus, using similar past as well as future contexts. We demonstrate that our automatically expanded reference sets lead to large improvements in correlations of automated metrics with human ratings of system outputs for DailyDialog dataset. 1 * VG and HJ contributed equally for this paper. Order decided by coin flip.

show abstract

Section: Related Workmentioning

confidence: 99%