DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization

Abstract

Dialogue is an essential part of human communication and cooperation. Existing research mainly focuses on short dialogue scenarios in a one-on-one fashion. However, multi-person interactions in the real world, such as meetings or interviews, are frequently over a few thousand words. There is still a lack of corresponding research and powerful tools to understand and process such long dialogues. Therefore, in this work, we present a pre-training framework for long dialogue understanding and summarization. Considering the nature of long conversations, we propose a window-based denoising approach for generative pre-training. For a dialogue, it corrupts a window of text with dialogue-inspired noise, and guides the model to reconstruct this window based on the content of the remaining conversation. Furthermore, to process longer input, we augment the model with sparse attention which is combined with conventional attention in a hybrid manner. We conduct extensive experiments on five datasets of long dialogues, covering tasks of dialogue summarization, abstractive question answering and topic segmentation. Experimentally, we show that our pre-trained model DialogLM significantly surpasses the state-of-the-art models across datasets and tasks. Source code and all the pre-trained models are available on our GitHub repository (https://github.com/microsoft/DialogLM).

TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning

“…Since the introduction of BERT (Devlin et al, 2019), the research community has witnessed remarkable progress in the field of language model pre-training on a large amount of free text. Such advancements have led to significant progresses in a wide range of natural language understanding (NLU) tasks Yang et al, 2019;Clark et al, 2020;Lan et al, 2021) and text generation tasks (Radford et al, 2019;Lewis et al, 2020;Raffel et al, 2020;Su et al, 2021a,e,g,d,f,c;Zhong et al, 2021) Contrastive Learning. Generally, contrastive learning methods distinguish observed data points from fictitious negative samples.…”

Exploring Neural Models for Query-Focused Summarization

“…Recent work on QMSum has introduced taskspecific denoising objectives for meeting summarization (Zhong et al, 2021a), generated final fine-grained summaries based on multiple coarsegrained steps (Zhang et al, 2021a), and treated the extractive text of an extractive-abstractive model as a latent variable (Mao et al, 2021). Zhang et al (2021b) analyze the challenges of long dialogue summarization such as the input length, the role of queries, and domain adaptation.…”

Abstractive Meeting Summarization: A Survey

“…For instance, BART proposes a full-text denoising pre-training objective for seq2seq models, whose cost is prohibitive for long text, and not tailored to dialogues. Zhong et al (2022) present the DialogLM model, along with a dialoguededicated, window-based denoising pretraining approach. Windows of consecutive utterances are first selected from the conversation, which is then disrupted with arbitrary dialogue-related noises, e.g., speaker mask, turn splitting, turn merging, text infilling and turn permutation.…”

“…As mentioned in Subsection 4.3, pretrained on free-form text makes these models difficult to process specifically structured dialogue transcriptions. Future research could follow the work of, notably DialoGPT (Zhang et al, 2020) and Di-alogLM (Zhong et al, 2022), to develop better languge models dedicated to spoken language, which will potentially offer large gains in task performance of abstractive meeting summarization. Commonsense incorporation.…”