Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset

Abstract

A significant barrier to progress in data-driven approaches to building dialog systems is the lack of high quality, goal-oriented conversational data. To help satisfy this elementary requirement, we introduce the initial release of the Taskmaster-1 dataset which includes 13,215 task-based dialogs comprising six domains. Two procedures were used to create this collection, each with unique advantages. The first involves a two-person, spoken "Wizard of Oz" (WOz) approach in which trained agents and crowdsourced workers interact to complete the task while the second is "selfdialog" in which crowdsourced workers write the entire dialog themselves. We do not restrict the workers to detailed scripts or to a small knowledge base and hence we observe that our dataset contains more realistic and diverse conversations in comparison to existing datasets. We offer several baseline models including state of the art neural seq2seq architectures with benchmark performance as well as qualitative human evaluations. Dialogs are labeled with API calls and arguments, a simple and cost effective approach which avoids the requirement of complex annotation schema. The layer of abstraction between the dialog model and the service provider API allows for a given model to interact with multiple services that provide similar functionally. Finally, the dataset will evoke interest in written vs. spoken language, discourse patterns, error handling and other linguistic phenomena related to dialog system research, development and design. 1 * Equal Contribution 1 Dataset available at https://g.co/dataset/taskmaster-1

Survey on evaluation methods for dialogue systems

“…Similar in size and content to MultiWOZ is Taskmaster-1 task-based dialogue dataset (Byrne et al 2019). It includes around 13K dialogues in six domains: ordering pizza, setting auto repair appointments, arranging taxi services, ordering movie tickets, ordering coffee drinks and making restaurant reservations.…”

Recent advances and challenges in task-oriented dialog systems

“…Some datasets contain singledomain conversations [48,51,67,68]. With the increasing demands to handle various tasks in real-world applications, some large-scale multi-domain corpora [69][70][71] have been collected recently. These datasets have higher language variation and task complexity.…”

Resource Constrained Dialog Policy Learning Via Differentiable Inductive Logic Programming

“…To foster research on dialog policy learning for virtual digital assistants, several task-oriented dialog corpora have been introduced in recent years, such as SimDial (Zhao and Eskenazi, 2018), Multi-WoZ (Budzianowski et al, 2018), Taskmaster (Byrne et al, 2019), and Schema Guided Dialog (Rastogi et al, 2019), to name a few. Deep learning approaches, including mixture models (Pei et al, 2019) hierarchical encoder/decoder Chen et al, 2019), reinforcement learning (Zhao et al, 2019), and pre-trained language models (Wu et al, 2019;Peng et al, 2020;Hosseini-Asl et al, 2020), have significantly advanced dialog policy research in the past few years , setting new state-of-the-art performance limits.…”