Can You Put it All Together: Evaluating Conversational Agents’ Ability to Blend Skills

Smith, Eric M.; Williamson, Mary; Shuster, Kurt; Weston, Jason; Boureau, Y-Lan

doi:10.18653/v1/2020.acl-main.183

Cited by 118 publications

(128 citation statements)

References 12 publications

Supporting

Mentioning

128

Contrasting

Order By: Relevance

“…Transformer models represent the state-of-the-art for many natural language processing (NLP) tasks, such as question-answering (Devlin et al, 2019), dialogue (Smith et al, 2020), search results (Nayak, 2019), and more. Popular pre-trained models, such as those available from Hugging Face (Wolf et al, 2019), allow developers without extensive computation power to benefit from these models.…”

Section: Introductionmentioning

confidence: 99%

Towards a Comprehensive Understanding and Accurate Evaluation of Societal Biases in Pre-Trained Transformers

Silva¹,

Tambwekar²,

Gombolay³

2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

The ease of access to pre-trained transformers has enabled developers to leverage large-scale language models to build exciting applications for their users. While such pre-trained models offer convenient starting points for researchers and developers, there is little consideration for the societal biases captured within these model risking perpetuation of racial, gender, and other harmful biases when these models are deployed at scale. In this paper, we investigate gender and racial bias across ubiquitous pre-trained language models, including GPT-2, XLNet, BERT, RoBERTa, ALBERT and Dis-tilBERT. We evaluate bias within pre-trained transformers using three metrics: WEAT, sequence likelihood, and pronoun ranking. We conclude with an experiment demonstrating the ineffectiveness of word-embedding techniques, such as WEAT, signaling the need for more robust bias testing in transformers.

show abstract

Section: Introductionmentioning

confidence: 99%

Towards a Comprehensive Understanding and Accurate Evaluation of Societal Biases in Pre-Trained Transformers

Silva¹,

Tambwekar²,

Gombolay³

2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

show abstract

“…Transformer-based chatbots, that are trained on a large amount of dialogue data [38], [64], are capable of generating a diverse range of responses. To evaluate such chatbots, our framework considers the specificity of dialogue responses within their corresponding contexts.…”

Section: Response Sampling Policymentioning

confidence: 99%

D-Score: Holistic Dialogue Evaluation Without Reference

Zhang

Lee

D’Haro

et al. 2021

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

In artistic gymnastics, difficulty score or D-score is used for judging performance. Starting from zero, an athlete earns points from different aspects such as composition requirement, difficulty, and connection between moves. The final score is a composition of the quality of various performance indicators. Similarly, when evaluating dialogue responses, human judges generally follow a number of criteria, among which language fluency, context coherence, logical consistency, and semantic appropriateness are on top of the agenda. In this paper, we propose an automatic dialogue evaluation framework called D-score that resembles the way gymnastics is evaluated. Following the four human judging criteria above, we devise a range of evaluation tasks and model them under a multi-task learning framework. The proposed framework, without relying on any human-written reference, learns to appreciate the overall quality of human-human conversations through a representation that is shared by all tasks without over-fitting to individual task domain. We evaluate D-score by performing comprehensive correlation analyses with human judgement on three dialogue evaluation datasets, among which two are from past DSTC series, and benchmark against state-of-the-art baselines. D-score not only outperforms the best baseline by a large margin in terms of system-level Spearman correlation but also represents an important step towards explainable dialogue scoring.

show abstract

“…Finally, human annotators label each of the remaining candidates as good or bad, with justifications (Section 2.3). We augment the Schema-Guided Dialogue (SGD) (Rastogi et al, 2020) and MultiWOZ 2.1 (Eric et al, 2020) corpora using the proposed approach. (See Figure 1 or Appendix A.4 for examples.)…”

Section: Introductionmentioning

confidence: 99%

Adding Chit-Chat to Enhance Task-Oriented Dialogues

Sun¹,

Moon²,

Crook³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Existing dialogue corpora and models are typically designed under two disjoint motives: while task-oriented systems focus on achieving functional goals (e.g., booking hotels), open-domain chatbots aim at making socially engaging conversations. In this work, we propose to integrate both types of systems by Adding Chit-Chat to ENhance Task-ORiented dialogues (ACCENTOR), with the goal of making virtual assistant conversations more engaging and interactive. Specifically, we propose a Human ↔ AI collaborative data collection approach for generating diverse chitchat responses to augment task-oriented dialogues with minimal annotation effort. We then present our new chit-chat-based annotations to 23.8K dialogues from two popular task-oriented datasets (Schema-Guided Dialogue and MultiWOZ 2.1) and demonstrate their advantage over the originals via human evaluation. Lastly, we propose three new models for adding chit-chat to task-oriented dialogues, explicitly trained to predict user goals and to generate contextually relevant chit-chat responses. Automatic and human evaluations show that, compared with the state-of-the-art task-oriented baseline, our models can codeswitch between task and chit-chat to be more engaging, interesting, knowledgeable, and humanlike, while maintaining competitive task performance.

show abstract

Can You Put it All Together: Evaluating Conversational Agents’ Ability to Blend Skills

Cited by 118 publications

References 12 publications

Towards a Comprehensive Understanding and Accurate Evaluation of Societal Biases in Pre-Trained Transformers

Towards a Comprehensive Understanding and Accurate Evaluation of Societal Biases in Pre-Trained Transformers

D-Score: Holistic Dialogue Evaluation Without Reference

Adding Chit-Chat to Enhance Task-Oriented Dialogues

Contact Info

Product

Resources

About