Human Annotated Dialogues Dataset for Natural Conversational Agents

Merdivan, Erinç; Singh, Deepika; Hanke, Sten; Kropf, Johannes; Holzinger, Andreas; Geist, Matthieu

doi:10.3390/app10030762

Cited by 13 publications

(12 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Lastly, the evaluation datasets should cover responses of a wide quality spectrum. In total, we have adopted six different publicly-available dialogue evaluation datasets with each accounting for a dialogue domain for assessing MDD-Eval 4 : DailyDialog-Eval (Zhao, Lala, and Kawahara 2020), Persona-Eval (Zhao, Lala, and Kawahara 2020), Topical-Eval (Mehri and Eskenazi 2020b), Movie-Eval (Merdivan et al 2020), Empathetic-Eval (Huang et al 2020 and Twitter-Eval (Hori and Hori 2017). Detailed statistics of each evaluation dataset is listed in Table 3.…”

Section: Evaluation Datasetsmentioning

confidence: 99%

“…However, the dialogue research heavily relies on the ability to evaluate system performance with automatic dialogue evaluation metrics (ADMs). Common natural language generation (NLG) metrics used in the dialogue system literature, such as BLEU (Papineni et al 2002) 2004), are unsuitable for the multi-domain dialogue evaluation task as they are shown to correlate poorly with human judgements (Liu et al 2016) due to the one-to-many contextresponse mapping in dialogues (Zhao, Zhao, and Eskenazi 2017) as well as the multi-faceted nature of dialogue evaluation (Mehri and Eskenazi 2020b). An alternative solution is to design model-based ADMs that explicitly learn to discriminate dialogue responses of varying quality.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation

Zhang

D’Haro

Friedrichs³

et al. 2022

AAAI

View full text Add to dashboard Cite

Chatbots are designed to carry out human-like conversations across different domains, such as general chit-chat, knowledge exchange, and persona-grounded conversations. To measure the quality of such conversational agents, a dialogue evaluator is expected to conduct assessment across domains as well. However, most of the state-of-the-art automatic dialogue evaluation metrics (ADMs) are not designed for multi-domain evaluation. We are motivated to design a general and robust framework, MDD-Eval, to address the problem. Specifically, we first train a teacher evaluator with human-annotated data to acquire a rating skill to tell good dialogue responses from bad ones in a particular domain and then, adopt a self-training strategy to train a new evaluator with teacher-annotated multi-domain data, that helps the new evaluator to generalize across multiple domains. MDD-Eval is extensively assessed on six dialogue evaluation benchmarks. Empirical results show that the MDD-Eval framework achieves a strong performance with an absolute improvement of 7% over the state-of-the-art ADMs in terms of mean Spearman correlation scores across all the evaluation benchmarks.

show abstract

Section: Evaluation Datasetsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation

Zhang

D’Haro

Friedrichs³

et al. 2022

AAAI

View full text Add to dashboard Cite

show abstract

“…• HUMOD-In the study [111], the unavailability of a common metric to evaluate the replies against human judgment is handled. This study contributes by developing a benchmark dataset with human annotation and diverse responses.…”

Section: Rq4 Which Techniques Are Effective For Reducing the Need For...mentioning

confidence: 99%

Framework for Deep Learning-Based Language Models Using Multi-Task Learning in Natural Language Understanding: A Systematic Literature Review and Future Directions

et al. 2022

View full text Add to dashboard Cite

Learning human languages is a difficult task for a computer. However, Deep Learning (DL) techniques have enhanced performance significantly for almost all-natural language processing (NLP) tasks. Unfortunately, these models cannot be generalized for all the NLP tasks with similar performance. NLU (Natural Language Understanding) is a subset of NLP including tasks, like machine translation, dialogue-based systems, natural language inference, text entailment, sentiment analysis, etc. The advancement in the field of NLU is the collective performance enhancement in all these tasks. Even though MTL (Multi-task Learning) was introduced before Deep Learning, it has gained significant attention in the past years. This paper aims to identify, investigate, and analyze various language models used in NLU and NLP to find directions for future research. The Systematic Literature Review (SLR) is prepared using the literature search guidelines proposed by Kitchenham and Charters on various language models between 2011 and 2021. This SLR points out that the unsupervised learning methodbased language models show potential performance improvement. However, they face the challenge of designing the generalpurpose framework for the language model, which will improve the performance of multi-task NLU and the generalized representation of knowledge. Combining these approaches may result in a more efficient and robust multi-task NLU. This SLR proposes building steps for a conceptual framework to achieve goals of enhancing the performance of language models in the field of NLU. INDEX TERMSDeep learning, Knowledge representation, Multi-task NLU, Unsupervised learningTABLE I APPLICATION DOMAINS FOR NLU Domain Applications Machine translation IBM Watson Task-based dialogue-based systems Booking tickets, taking an appointment using Google assistant

show abstract

“…To fill the above gap, we curate a novel largescale silver dialogue dataset, EDOS (Emotional Dialogues in OpenSubtitles), containing 1M emotional dialogues from movie subtitles, in which each dialogue turn is automatically annotated with 32 fine-grained emotions, eight plus categories as well as the Neutral category. Movie subtitles are extensively used for emotion analysis in text in earlier and recent research (Kayhani et al, 2020;Merdivan et al, 2020;Giannakopoulos et al, 2009). The Nature article "How movies mirror our mimicry" (Ball, 2011) states "screenwriters mine everyday discourse to make dialogues appear authentic" and "audiences use language devices in movies to shape their own discourse".…”

Section: Speakermentioning

confidence: 99%

A Large-Scale Dataset for Empathetic Response Generation

Welivita¹,

Xie²,

Pu³

2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Recent development in NLP shows a strong trend towards refining pre-trained models with a domain-specific dataset. This is especially the case for response generation where emotion plays an important role. However, existing empathetic datasets remain small, delaying research efforts in this area, for example, the development of emotion-aware chatbots. One main technical challenge has been the cost of manually annotating dialogues with the right emotion labels. In this paper, we describe a large-scale silver dataset consisting of 1M dialogues annotated with 32 fine-grained emotions, eight empathetic response intents, and the Neutral category. To achieve this goal, we have developed a novel data curation pipeline starting with a small seed of manually annotated data and eventually scaling it to a satisfactory size. We compare its quality against a state-of-the-art gold dataset using offline experiments and visual validation methods. The resultant procedure can be used to create similar datasets in the same domain as well as in other domains. 1

show abstract

Human Annotated Dialogues Dataset for Natural Conversational Agents

Cited by 13 publications

References 25 publications

MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation

MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation

Framework for Deep Learning-Based Language Models Using Multi-Task Learning in Natural Language Understanding: A Systematic Literature Review and Future Directions

A Large-Scale Dataset for Empathetic Response Generation

Contact Info

Product

Resources

About