Knowledge-Grounded Dialogue Generation with Pre-trained Language Models

Zhao, Xueliang; Wu, Wei; Xu, Can; Tao, Chongyang; Zhao, Dongyan; Yan, Rui

doi:10.18653/v1/2020.emnlp-main.272

Cited by 126 publications

(161 citation statements)

References 53 publications

Supporting

Mentioning

161

Contrasting

Order By: Relevance

“…Automated and human evaluation results on three different datasets demonstrate substantial improvements ( §5.1 and §5.2). Specifically, we achieve an improvement of 19.7 BLEU-4 points compared to Zhao et al (2020b) on the dialogue generation task. Additionally, significant gains are observed in BLEU-4 compared to BART-based baseline.…”

Section: Introductionmentioning

confidence: 90%

“…A dialogue manager is used to combine the vocabulary distributions provided by these three components. Zhao et al (2020b) propose a knowledge selection module integrated with pre-trained language models for this task. Cao et al (2020) use pre-trained language model GPT-2 (Radford et al) and explore various attention fusion techniques for persona-based dialogue generation (Zhang et al, 2018b;.…”

Section: Related Workmentioning

confidence: 99%

“…Our main contributions are the two new proposed techniques for the document grounded generation tasks ( §3.2 and §3.3). We also provide a new baseline which is stronger than the previous state-of-the-art methods (Zhao et al, 2020b;Prabhumoye et al, 2019) for the two tasks. We formally show how the two independent tasks studied in this paper are identical and similar modeling techniques can be used to solve them ( §3).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Focused Attention Improves Document-Grounded Generation

Prabhumoye¹,

Hashimoto²,

Zhou³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Document grounded generation is the task of using the information provided in a document to improve text generation. This work focuses on two different document grounded generation tasks: Wikipedia Update Generation task and Dialogue response generation. Our work introduces two novel adaptations of large scale pre-trained encoder-decoder models focusing on building context driven representation of the document and enabling specific attention to the information in the document. Additionally, we provide a stronger BART baseline for these tasks. Our proposed techniques outperform existing methods on both automated (at least 48% increase in BLEU-4 points) and human evaluation for closeness to reference and relevance to the document. Furthermore, we perform comprehensive manual inspection of the generated output and categorize errors to provide insights into future directions in modeling these tasks.

show abstract

Section: Introductionmentioning

confidence: 90%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Focused Attention Improves Document-Grounded Generation

Prabhumoye¹,

Hashimoto²,

Zhou³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

show abstract

“…To address this issue, a line of research proposes to incorporate external knowledge into the generation process. Most of the work in this line retrieves knowledge based on a search or retrieval step first, and followed by further reranking of retrieved relevant knowledge snippets (Ghazvininejad et al, 2018;Young et al, 2018;Zhou et al, 2018b;Gopalakrishnan et al, 2019;Zhao et al, 2020). In our work, we propose neural entity recognition and linking to identify and resolve entities more accurately in order to obtain more relevant knowledge for knowledge grounded response generation.…”

Section: The First Two Authors Have Equal Contributionmentioning

confidence: 99%

Entity Resolution in Open-domain Conversations

Shang¹,

Wang²,

Eric³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

In recent years, incorporating external knowledge for response generation in open-domain conversation systems has attracted great interest. To improve the relevance of retrieved knowledge, we propose a neural entity linking (NEL) approach. Different from formal documents such as news, conversational utterances are informal and multi-turn, which makes it more challenging to disambiguate the entities. Therefore, we present a context-aware named entity recognition model (NER) and entity resolution (ER) model to utilize dialogue context information. We conduct NEL experiments on three open-domain conversation datasets and validate that incorporating context information improves the performance of NER and ER models. Furthermore, we verify that using knowledge sentences identified based on NEL benefits the neural response generation model.

show abstract

“…Research in dialogue generation has rapidly evolved from sequence-to-sequence (Sutskever et al, 2014) and Transformer models (Vaswani et al, 2017) to approaches with pre-trained models such as BERT (Devlin et al, 2019), XLNet and T5 (Raffel et al, 2020). More recently, it included techniques that use knowledge, in addition to the original posts, to improve the quality of the generated responses (Ghazvininejad et al (2018), Moghe et al (2018), Dinan et al (2019), Galley et al (2019), Lian et al (2019), Zheng and Zhou (2019), Zhao et al (2020a), Zhao et al (2020b)). 1 This approach is referred to as…”

Section: Introductionmentioning

confidence: 99%

Knowledge-Grounded Dialogue Generation with Term-level De-noising

Zheng¹,

Milić-Frayling

Zhou

2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

Dialogue generation has been improvedthrough injecting knowledge into generative models. However, addition of knowledge through simple selection of sentences or paragraphs is likely to introduce noise and diminish the effectiveness of the generative models. In this paper, we present a novel Knowledge Term Weighting Model (KTWM) that incorporates term-level de-noising of the selected knowledge. KTWM includes a module for generating Simulated Response Vectors (SRVs) and uses SRVs attention distributions with the knowledge embeddings to determine knowledge term weights. Our experiments demonstrate that KTWM, combined with various knowledge selection algorithms, consistently achieves statistically significant improvements over methods without term weighting when applied to two publicly available datasets Wizard of Wikipedia (Wiz) and Holl-E. The results are particularly improved for the Wiz test data with unseen topics, demonstrating the robustness of the KTWM noise-reduction approach.

show abstract

Knowledge-Grounded Dialogue Generation with Pre-trained Language Models

Cited by 126 publications

References 53 publications

Focused Attention Improves Document-Grounded Generation

Focused Attention Improves Document-Grounded Generation

Entity Resolution in Open-domain Conversations

Knowledge-Grounded Dialogue Generation with Term-level De-noising

Contact Info

Product

Resources

About