Multi-Document Summarization with Determinantal Point Processes and Contextualized Representations

Abstract

Emerged as one of the best performing techniques for extractive summarization, determinantal point processes select the most probable set of sentences to form a summary according to a probability measure defined by modeling sentence prominence and pairwise repulsion. Traditionally, these aspects are modelled using shallow and linguistically informed features, but the rise of deep contextualized representations raises an interesting question of whether, and to what extent, contextualized representations can be used to improve DPP modeling. Our findings suggest that, despite the success of deep representations, it remains necessary to combine them with surface indicators for effective identification of summary sentences.

Better Highlighting: Creating Sub-Sentence Summary Highlights

“…Table 7 presents a direct comparison of XLNet and tree segments on DUC and TAC datasets. We find that XLNet segments are more concise than 5 Our preliminary experiment comparing the quality classifier against that of Cho et al (2019b) shows that our method obtains significantly better classification accuracy (70% vs. tree segments. A tree segment contains 13 tokens on average, while an XLNet segment contains 9.6 tokens on DUC-04.…”

“…We compare our method with strong extractive and abstractive summarization systems for multidocument summarization, results are shown in Tables 3 and 5. DPP (Kulesza and Taskar, 2012) and variant DPP-BERT (Cho et al, 2019b) use determinantal point processes to extract whole sentences from a set of documents. SumBasic (Vanderwende et al, 2007) is an extractive approach leveraging the fact that frequently occurring words are more likely to be included in the summary.…”

“…(1). Estimating the pairwise similarity S is trivial, we refer the reader to (Cho et al, 2019b) for details. In this paper, we present a inverted pyramid method to estimate the quality of segments q.…”

“…DPP-BERT (Cho et al, 2019b) 39.05 10.23 14.35 DPP (Kulesza and Taskar, 2012) 38.10 9.14 13.40 SumBasic (Vanderwende et al, 2007) 29.48 4.25 8.64 KLSumm (Haghighi et al, 2009) 31.04 6.03 10.23 LexRank (Erkan and Radev, 2004) 34.44 7.11 11.19 Centroid (Hong et al, 2014) 35.49 7.80 12.02 ICSISumm (Gillick and Favre, 2009) 37.31 9.36 13.12 Opinosis (Ganesan et al, 2010) 27.07 5.03 8.63 Pointer-Gen (See et al, 2017) 31.43 6.03 10.01 CopyTrans (Gehrmann et al, 2018) 28.54 6.38 7.22 Hi-MAP (Fabbri et al, 2019) 35…”

“…R-1 R-2 R-SU4 DPP-BERT (Cho et al, 2019b) 38.59 11.06 14.65 DPP (Kulesza and Taskar, 2012) 36.95 9.83 13.57 SumBasic (Vanderwende et al, 2007) 31.58 6.06 10.06 KLSumm (Haghighi et al, 2009) 31.23 7.07 10.56 LexRank (Erkan and Radev, 2004) 33.10 7.50 11.13 Opinosis (Ganesan et al, 2010) 25.15 5.12 8.12 Pointer-Gen (See et al, 2017) 31 the training data. The maximum sequence length of the model is 512 tokens and the batch size is set to 16.…”