PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable

Abstract

Pre-training models have been proved effective for a wide range of natural language processing tasks. Inspired by this, we propose a novel dialogue generation pre-training framework to support various kinds of conversations, including chitchat , knowledge grounded dialogues, and conversational question answering. In this framework, we adopt flexible attention mechanisms to fully leverage the bi-directional context and the uni-directional characteristic of language generation. We also introduce discrete latent variables to tackle the inherent one-to-many mapping problem in response generation. Two reciprocal tasks of response generation and latent act recognition are designed and carried out simultaneously within a shared network. Comprehensive experiments on three publicly available datasets verify the effectiveness and superiority of the proposed framework.

TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue

“…Con-veRT (Henderson et al, 2019a) pre-trained a dual transformer encoder for response selection task on large-scale Reddit (input, response) pairs. PLATO (Bao et al, 2019) uses both Twitter and Reddit data to pre-trained a dialogue generation model with discrete latent variables. All of them are designed to cope with the response generation task for opendomain chatbots.…”

“…However, previous work (Rashkin et al, 2018;Wolf et al, 2019) shows that there are some deficiencies in the performance to apply fine-tuning on conversational corpora directly. One possible reason could be the intrinsic difference of linguistic patterns between human conversations and writing text, resulting in a large gap of data distributions (Bao et al, 2019). Therefore, pre-training dialogue language models using chit-chat corpora from social media, such as Twitter or Reddit, has been recently investigated, especially for dialogue response generation (Zhang et al, 2019) and retrieval (Henderson et al, 2019b).…”

Filtering Noisy Dialogue Corpora by Connectivity and Content Relatedness

“…See Appendix E.5 for more extensive evaluation results.15 Same size asShen et al (2017) andBao et al (2020).16 See Appendix F for all experimental results on Japanese.…”

Simple Data Augmentation with the Mask Token Improves Domain Adaptation for Dialog Act Tagging

“…2019; Zhang et al, 2019;Bao et al, 2020;Henderson et al, 2019;. However, domain adaptation capabilities of these models remain to be further explored for goal-oriented dialogues.…”