Planning with Learned Entity Prompts for Abstractive Summarization

Abstract

We introduce a simple but flexible mechanism to learn an intermediate plan to ground the generation of abstractive summaries. Specifically, we prepend (or prompt) target summaries with entity chains—ordered sequences of entities mentioned in the summary. Transformer-based sequence-to-sequence models are then trained to generate the entity chain and then continue generating the summary conditioned on the entity chain and the input. We experimented with both pretraining and finetuning with this content planning objective. When evaluated on CNN/DailyMail, XSum, SAMSum, and BillSum, we demonstrate empirically that the grounded generation with the planning objective improves entity specificity and planning in summaries for all datasets, and achieves state-of-the-art performance on XSum and SAMSum in terms of rouge. Moreover, we demonstrate empirically that planning with entity chains provides a mechanism to control hallucinations in abstractive summaries. By prompting the decoder with a modified content plan that drops hallucinated entities, we outperform state-of-the-art approaches for faithfulness when evaluated automatically and by humans.

Faithful to the Document or to the World? Mitigating Hallucinations via Entity-linked Knowledge in Abstractive Summarization

“…aspects: faithful to the source document (extractiveness) and faithful to the world knowledge (abstractiveness). Contrary to previous work that improves factual consistency by filtering training examples to contain only extractive entities (Nan et al, 2021;Narayan et al, 2021b;Mao et al, 2020), we focus on improving the faithfulness of the generated entities from the abstractive perspective: by providing additional facts that are relevant to the source.…”

“…Recently, many methods have also been proposed to improve the factual consistency of generated summaries. The majority of these models reduces the probability of generating novel entities by imposing constraint w.r.t the source, such as quantity entity matching (Zhao et al, 2020), intermediate planning with entity chains (Narayan et al, 2021b), or simple filtering (Nan et al, 2021). Filippova (2020) controls hallucinations with unconditional and conditional LMs.…”

“…In this paper, we experiment with incorporating facts that are directly linked to the entities in the source. Several models have been proposed to combine symbolically interpretable factual information and subsymbolic neural knowledge Narayan et al, 2021a).…”

FRUIT: Faithfully Reflecting Updated Information in Text

“…Second, before each update we insert a sequence of reference tokens identifying the pieces of evidence that support the update, e.g., if the first and third piece of evidence in E t→t support an update then the update is prefaced by (1)(3). This approach, inspired by the use of entity chains for summarization (Narayan et al, 2021), trains the model to plan which references to use before generating an update. These reference tokens are removed from the output text of the model prior to computing the evaluation metrics.…”

“…In this section we investigate whether ad- ditional control provided by the users can improve the overall generations. We follow Keskar et al (2019) and Narayan et al (2021), and provide more detailed instruction by adding control codes, i.e., special tokens, to the input that instruct the model whether to add, copy, edit or remove a sentence, as well as which evidence to use when making an addition or edit. We use the target text to provide the oracle labels for the control code, and see if the EDIT5 can take advantages of the codes to generate better output.…”