TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents

Wolf, Thomas; Sanh, Victor; Chaumond, Julien; Delangue, Clément

doi:10.48550/arxiv.1901.08149

Cited by 134 publications

(225 citation statements)

References 15 publications

Supporting

Mentioning

221

Contrasting

Order By: Relevance

“…Output: Dialogue Response Persona [Li et al 2016b;Mazaré et al 2018;Siddique et al 2017;Wolf et al 2019] [ Song et al , 2020bZhang et al 2018b;Zhong et al 2020], [Zeng and Nie 2021;Zheng et al 2020] Politeness [Niu and Bansal 2018],…”

Section: Transformer-based Pre-trained Language Modelsmentioning

confidence: 99%

A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models

Zhang¹,

Song²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Controllable Text Generation (CTG) is emerging area in the field of natural language generation (NLG). It is regarded as crucial for the development of advanced text generation technologies that are more natural and better meet the specific constraints in practical applications. In recent years, methods using large-scale pre-trained language models (PLMs), in particular the widely used transformer-based PLMs, have become a new paradigm of NLG, allowing generation of more diverse and fluent text. However, due to the lower level of interpretability of deep neural networks, the controllability of these methods need to be guaranteed. To this end, controllable text generation using transformer-based PLMs has become a rapidly growing yet challenging new research hotspot. A diverse range of approaches have emerged in the recent 3-4 years, targeting different CTG tasks which may require different types of controlled constraints. In this paper, we present a systematic critical review on the common tasks, main approaches and evaluation methods in this area. Finally, we discuss the challenges that the field is facing, and put forward various promising future directions. To the best of our knowledge, this is the first survey paper to summarize CTG techniques from the perspective of PLMs. We hope it can help researchers in related fields to quickly track the academic frontier, providing them with a landscape of the area and a roadmap for future research.

show abstract

Section: Transformer-based Pre-trained Language Modelsmentioning

confidence: 99%

A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models

Zhang¹,

Song²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Full Objectives. The entire loss function aims to minimize the negative log-likelihood of language modeling and subtasks as in (Radford et al 2018;Wolf et al 2019). The full training objectives are defined as follows:…”

Section: Dialog Modulementioning

confidence: 99%

Call for Customized Conversation: Customized Conversation Grounding Persona and Knowledge

Jang¹,

Lim²,

Hur³

et al. 2021

Preprint

View full text Add to dashboard Cite

Humans usually have conversations by making use of prior knowledge about a topic and background information of the people whom they are talking to. However, existing conversational agents and datasets do not consider such comprehensive information, and thus they have a limitation in generating the utterances where the knowledge and persona are fused properly. To address this issue, we introduce a call For Customized conversation (FoCus) dataset where the customized answers are built with the user's persona and Wikipedia knowledge. To evaluate the abilities to make informative and customized utterances of pre-trained language models, we utilize BART and GPT-2 as well as transformer-based models. We assess their generation abilities with automatic scores and conduct human evaluations for qualitative results. We examine whether the model reflects adequate persona and knowledge with our proposed two sub-tasks, persona grounding (PG) and knowledge grounding (KG). Moreover, we show that the utterances of our data are constructed with the proper knowledge and persona through grounding quality assessment.

show abstract

“…Therefore, with the success of pretrained language models (PrLMs) like BERT (Devlin et al 2019), RoBERTa (Liu et al 2019), a series of neural approaches based on crossencoders are proposed (Vig and Ramea 2019;Wolf et al 2019). Although enjoying satisfying retrieval rate, the retrieval time latency is often hard to tolerant in practical use.…”

Section: Open-domain Passage Retrievalmentioning

confidence: 99%

Sentence-aware Contrastive Learning for Open-Domain Passage Retrieval

Bo-hong¹,

Zhang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Training dense passage representations via contrastive learning (CL) has been shown effective for Open-Domain Passage Retrieval (ODPR). Recent studies mainly focus on optimizing this CL framework by improving the sampling strategy or extra pretraining. Different from previous studies, this work devotes itself to investigating the influence of conflicts in the widely used CL strategy in ODPR, motivated by our observation that a passage can be organized by multiple semantically different sentences, thus modeling such a passage as a unified dense vector is not optimal. We call such conflicts Contrastive Conflicts. In this work, we propose to solve it with a representation decoupling method, by decoupling the passage representations into contextual sentence-level ones, and design specific CL strategies to mediate these conflicts. Experiments on widely used datasets including Natural Questions, Trivia QA, and SQuAD verify the effectiveness of our method, especially on the dataset where the conflicting problem is severe. Our method also presents good transferability across the datasets, which further supports our idea of mediating Contrastive Conflicts.

show abstract

TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents

Cited by 134 publications

References 15 publications

A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models

A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models

Call for Customized Conversation: Customized Conversation Grounding Persona and Knowledge

Sentence-aware Contrastive Learning for Open-Domain Passage Retrieval

Contact Info

Product

Resources

About