TicketTalk: Toward human-level performance with end-to-end, transaction-based dialog systems

Abstract

We present a data-driven, end-to-end approach to transaction-based dialog systems that performs at near-human levels in terms of verbal response quality and factual grounding accuracy. We show that two essential components of the system produce these results: a sufficiently large and diverse, in-domain labeled dataset, and a neural network-based, pretrained model that generates both verbal responses and API call predictions. In terms of data, we introduce TicketTalk, a movie ticketing dialog dataset with 23,789 annotated conversations. The movie ticketing conversations range from completely open-ended and unrestricted to more structured, both in terms of their knowledge base, discourse features, and number of turns. In qualitative human evaluations, model-generated responses trained on just 10,000 TicketTalk dialogs were rated to "make sense" 86.5% of the time, almost the same as human responses in the same contexts. Our simple, API-focused annotation schema results in a much easier labeling task making it faster and more cost effective. It is also the key component for being able to predict API calls accurately. We handle factual grounding by incorporating API calls in the training data, allowing our model to learn which actions to take and when. Trained on the same 10,000-dialog set, the model's API call predictions were rated to be correct 93.9% of the time in our evaluations, surpassing the ratings for the corresponding human labels. We show how API prediction and response generation scores improve as the dataset size incrementally increases from 5000 to 21,000 dialogs. Our analysis also clearly illustrates the benefits of pre-training. To facilitate future work on transaction-based dialog systems, we have published the TicketTalk dataset at https:// git.io/JL8an.

Language modeling via stochastic processes

“…The assistant's turn is marked with an "[ASSISTANT]" tag, and the user's turn is marked with a "[USER]" tag. TicketTalk (Byrne et al, 2021) contains conversations on booking movie tickets between an assistant and a user. The assistant's and user's turns are similarly marked as in TM-2.…”

“…TicketTalk (Byrne et al, 2021) which contains conversations on booking movie tickets between an assistant and a user. The assistant's and user's turned are similarly marked as in TM-2.…”

A Unified Multi-task Learning Framework for Multi-goal Conversational Recommender Systems

“…As mentioned in Section 2.2, recent state-of-the-art performance [16,21] on dialogue state tracking is achieved by the reformulation of such a structure prediction task as a text generation task. In addition, Byrne et al [3] investigate the unified Seq2Seq paradigm for action prediction in transaction-based dialogue systems. However, there are more complicated subtasks with diverse paradigms in MG-CRS, varying from multi-label classification, ranking/recommendation, to text generation.…”

Helpfulness and Fairness of Task-Oriented Dialogue Systems

“…Therefore, we use task-oriented dialogue data as an proxy. We choose TaskMaster (Byrne et al, 2019(Byrne et al, , 2021 over other task-oriented datasets such as synthesis datasets (e.g., MultiWOZ (Budzianowski et al, 2018)) and datasets for scheduling (e.g., KVRET (Eric et al, 2017)), because TaskMaster is collected from real human dialogue and contains a more diverse set of scenarios including restaurant suggestions, movie recommendations, flight reservations, and etc.…”