Enriching and Controlling Global Semantics for Text Summarization

Abstract

Recently, Transformer-based models have been proven effective in the abstractive summarization task by creating fluent and informative summaries. Nevertheless, these models still suffer from the short-range dependency problem, causing them to produce summaries that miss the key points of document. In this paper, we attempt to address this issue by introducing a neural topic model empowered with normalizing flow to capture the global semantics of the document, which are then integrated into the summarization model. In addition, to avoid the overwhelming effect of global semantics on contextualized representation, we introduce a mechanism to control the amount of global semantics supplied to the text generation module. Our method outperforms state-of-the-art summarization models on five common text summarization datasets, namely CNN/DailyMail, XSum, Reddit TIFU, arXiv, and PubMed.

Boosting Summarization with Normalizing Flows and Aggressive Training

“…Our model belongs to the first category, akin to prior studies like Setiawan et al (2020);Fu et al (2020). In contrast, other works, including Zheng et al (2020); Nguyen et al (2021); Zou et al (2021), adopt the second type by jointly modeling topics and sequence-to-sequence generation. Most of them assume a simple Gaussian latent prior, except for Nguyen et al (2021), which employs normalizing flows to model neural topic models and enrich global semantics.…”

“…In contrast, other works, including Zheng et al (2020); Nguyen et al (2021); Zou et al (2021), adopt the second type by jointly modeling topics and sequence-to-sequence generation. Most of them assume a simple Gaussian latent prior, except for Nguyen et al (2021), which employs normalizing flows to model neural topic models and enrich global semantics. However, they did not specify the choice of normalizing flows and how they addressed posterior collapse.…”

Pretrained Language Models for Text Generation: A Survey

“…Therefore, it is important to retain the most critical contents of documents and verbalize them in the generated text. To address the issue of key points missing in output text, Nguyen et al [134] introduced a topic model to capture the global semantics of the document and a mechanism to control the amount of global semantics supplied to the text generation module. Similarly, Liu et al [114] also proposed two topic-aware contrastive learning objectives to capture the global topic information of a conversation and outline salient facts.…”

“…CIT [162] employs an extractor (RoBERTa) to extract the important words and sentences from the input, which will be fed into encoder with the input. In addition, topic models can capture the global semantics of the document, which can integrated into the summarization model [134]. Finally, GSum [36] proposes a general framework taking different kinds of guidance into the generation model, including keywords, triples, highlighted sentences and retrieved summaries.…”

“…RL: DialogBERT[61]; Li et al[109]; Graph-based RL: BASS[191]; Ouyang et al[136] Document Encoding Inter-Sentential Semantics: BERTSumExt[119]; HIBERT et al[219]; Capturing Critical Semantics: Nguyen et al[134]; Liu et al[114]; RL Efficiency: Huang et al…”