TODSum: Task-Oriented Dialogue Summarization with State Tracking

Abstract

Previous dialogue summarization datasets mainly focus on open-domain chitchat dialogues, while summarization datasets for the broadly used task-oriented dialogue haven't been explored yet. Automatically summarizing such taskoriented dialogues can help a business collect and review needs to improve the service. Besides, previous datasets pay more attention to generate good summaries with higher ROUGE scores, but they hardly understand the structured information of dialogues and ignore the factuality of summaries. In this paper, we introduce a large-scale public Task-Oriented Dialogue Summarization dataset, TODSum, which aims to summarize the key points of the agent completing certain tasks with the user. Compared to existing work, TODSum suffers from severe scattered information issues and requires strict factual consistency, which makes it hard to directly apply recent dialogue summarization models. Therefore, we introduce additional dialogue state knowledge for TODSum to enhance the faithfulness of generated summaries. We hope a better understanding of conversational content helps summarization models generate concise and coherent summaries. Meanwhile, we establish a comprehensive benchmark for TODSum and propose a state-aware structured dialogue summarization model to integrate dialogue state information and dialogue history. Exhaustive experiments and qualitative analysis prove the effectiveness of dialogue structure guidance. Finally, we discuss the current issues of TODSum and potential development directions for future work.

Instructive Dialogue Summarization with Query Aggregations

“…Dialogue summarization is the task of generating a concise and fluent summary of a conversation involving two or more participants. It has gained significant attention due to its broad applications and availability of relevant datasets (Gliwa et al, 2019;Zhao et al, 2021). Solutions on dialogue summarization are mainly based on sequence-to-sequence models including the pointer-generation network (See et al, 2017), T5 (Raffel et al, 2020) and BART (Lewis et al, 2020).…”

“…For length awareness, each sample is augmented once. This results in a total of (14.7k + 12.5k + We evaluate and benchmark our method on three dialogue summarization datasets including SAM-Sum (Gliwa et al, 2019), DialogSum and TODSum (Zhao et al, 2021). These datasets are equipped with dialogues and human-written or verified summaries.…”

Multi-User MultiWOZ: Task-Oriented Dialogues among Multiple Users

“…Our dataset focuses on summaries where deliberations are resolved and only task-relevant information (e.g., user intents and slots) is retained. While some datasets aim to summarize task-oriented dialogues from customer services (Zhao et al, 2021;Feigenblat et al, 2021;Lin et al, 2021), they are not designed to summarize chats between users; rather, they summarize a chat between user and agent, which has different dynamics than collaborative decision-making between users.…”