Self-Attention Guided Copy Mechanism for Abstractive Summarization

Abstract

Copy module has been widely equipped in the recent abstractive summarization models, which facilitates the decoder to extract words from the source into the summary. Generally, the encoder-decoder attention is served as the copy distribution, while how to guarantee that important words in the source are copied remains a challenge. In this work, we propose a Transformer-based model to enhance the copy mechanism. Specifically, we identify the importance of each source word based on the degree centrality with a directed graph built by the self-attention layer in the Transformer. We use the centrality of each source word to guide the copy process explicitly. Experimental results show that the self-attention graph provides useful guidance for the copy distribution. Our proposed models significantly outperform the baseline methods on the CNN/Daily Mail dataset and the Gigaword dataset.

Focus Attention: Promoting Faithfulness and Diversity in Summarization

“…We see similar kind of improvements as observed in Table 1, except for ROUGE-2 for ROBFAME which is 0.23 points worse than the ROBERTAS2S baseline. Our best model PEG-FAME performs better than both copy mechanism models: LSTM-based PtGen (See et al, 2017) and Transformer-based SAGCopy (Xu et al, 2020). PEGFAME performs worse when compared with T5 (Raffel et al, 2019), the original PEGASUS and ProphetNet (Qi et al, 2020).…”

“…Cao et al (2017) force faithful generation by conditioning on both source text and extracted fact descriptions from the source text. Song et al (2020) propose to jointly generate a sentence and its syntactic dependency parse to induce grammaticality and faithfulness. Tian et al (2019) learn a confidence score to ensure that the model attends to the source whenever necessary.…”

Highlight-Transformer: Leveraging Key Phrase Aware Attention to Improve Abstractive Multi-Document Summarization

“…Previous encoder-decoder models (Rush et al, 2015;Nallapati et al, 2016;Paulus et al, 2018;Chopra et al, 2016) equipped with the attention mechanism (Bahdanau et al, 2015) have achieved great performance on abstractive summarization. However, they were found to miss some important content in input documents (Li et al, 2018;Xu et al, 2020). How to retain the key information of input documents in the generated summaries has received increasing attention in the past few years.…”

“…Gehrmann et al (2018) utilize the attention masks to restrict copying phrases from the selected parts of an input document. Xu et al (2020) explicitly guide the copy process with the centrality of each source word. Several papers also explore the potential of enhancing the encoder.…”

“…The attention distribution of tokens is multiplied by the MMR score of the sentence to which they belong. SAGCopy (Xu et al, 2020) adds the word centrality score to the linearly transformed hidden state when calculating the copy distribution.…”