Focused Attention Improves Document-Grounded Generation

Abstract

Document grounded generation is the task of using the information provided in a document to improve text generation. This work focuses on two different document grounded generation tasks: Wikipedia Update Generation task and Dialogue response generation. Our work introduces two novel adaptations of large scale pre-trained encoder-decoder models focusing on building context driven representation of the document and enabling specific attention to the information in the document. Additionally, we provide a stronger BART baseline for these tasks. Our proposed techniques outperform existing methods on both automated (at least 48% increase in BLEU-4 points) and human evaluation for closeness to reference and relevance to the document. Furthermore, we perform comprehensive manual inspection of the generated output and categorize errors to provide insights into future directions in modeling these tasks.

FaithDial: A Faithful Benchmark for Information-Seeking Dialogue

“…What is more, Dziri et al (2022) observe that GPT2 not only replicates, but even amplifies hallucination around 20% when trained on WOW. This finding also extends to models that are designed explicitly to be knowledgegrounded (Prabhumoye et al, 2021;. Filtering noisy or high-error data (Zhang and Hashimoto, 2021) is also prone to failure as it may either break the cohesion of discourse or it may require excluding entire dialogues.…”

“…Despite the recent success of knowledge-grounded neural conversational models (Thoppilan et al, 2022;Prabhumoye et al, 2021;Zhao et al, 2020, inter alia) in generating fluent responses, they also generate unverifiable or factually incorrect statements, a phenomenon known as hallucinations Broken heart is a term metaphor for the intense emotional and sometimes physical stress or pain one feels at experiencing great longing.…”

Multi-Stage Prompting for Knowledgeable Dialogue Generation

“…For evaluating both knowledge generation and response generation, we follow previous works Dinan et al, 2018;Prabhumoye et al, 2021) to evaluate the generated sentences against the reference sentences on averaged BLEU (an average of BLEU-1,2,3,4) (Papineni et al, 2002), ROUGE-L (Lin, 2004), METEOR (Denkowski and Lavie, 2011), and unigram F1. Additionally, we follow Komeili et al (2021) to use knowledge F1 (KF1) to evaluate the knowledgeability of the response generation.…”

“…Grounding dialogue responses based on a knowledge base ensures a knowledgeable and engaging response and is emerging as an important step in research of human-machine conversation (Zhu et al, 2017;Ghazvininejad et al, 2018;Dinan et al, 2018;Zhou et al, 2018;Kim et al, 2019;Moon et al, 2019;Zhao et al, 2019;Li et al, 2020;Hedayatnia et al, 2020;Zhan et al, 2021;Prabhumoye et al, 2021;Rashkin et al, 2021;Komeili et al, 2021). Kim et al (2019) proposed sequential knowledge transformer to boost the knowledge selection quality from the candidates, and improved the performance of the response generation.…”

“…Dialogue systems face the problem of producing bland and generic outputs that are devoid of content (Wolf et al, 2019;Holtzman et al, 2019;Ma et al, 2020). Recent efforts have been made to address these concerns by grounding dialogue responses on a source of knowledge (Dinan et al, 2018; Zhou et al, 2018;Zhao et al, 2019;Santhanam et al, 2020;Prabhumoye et al, 2021). Therefore, building a knowledgeable dialogue system has become one of the key milestone tasks in conversational research.…”