Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization

Abstract

Text summarization aims to generate a short summary for an input text. In this work, we propose a Non-Autoregressive Unsupervised Summarization (NAUS) approach, which does not require parallel data for training. Our NAUS first performs edit-based search towards a heuristically defined score, and generates a summary as pseudo-groundtruth. Then, we train an encoder-only non-autoregressive Transformer based on the search result. We also propose a dynamic programming approach for length-control decoding, which is important for the summarization task. Experiments on two datasets show that NAUS achieves state-of-the-art performance for unsupervised summarization, yet largely improving inference efficiency. Further, our algorithm is able to perform explicit length-transfer summary generation. 1

Hierarchical Latent Alignment for Non-Autoregressive Generation under High Compression Ratio

“…CTC-based models have shown advantages in handling token repetition and eliminating the need for length prediction [8], [11] compared with the other non-autoregressive models [4], [25]. CTC-based models support an encoder-only architecture and have been used in machine translation [8], [11] and sentence summarization [5], [6].…”

“…CTC-based models have also been adapted to sentence summarization. Liu et al [5] address unsupervised sentence summarization and use the CTC-based model to learn from the search results [34]. Liu et al [6] develop a character-level length-control algorithm for sentence summarization based on CTC.…”

“…In recent years, there has been an increasing interest in applying non-autoregressive models to summarization [5], [25]. On the one hand, the non-autoregressive models are faster than the autoregressive generation, which is important for real-world applications [5]. On the other hand, the non-autoregressive models support the encoder-only architecture [5].…”

“…Non-autoregressive generation (NAR) models generate each token simultaneously [1]- [4] and achieve fast decoding speed, which is important in real-world applications. Moreover, parallel decoding brings convenience to control the output by designing the decoding algorithm [5], [6]. Thus NAR has attracted more and more attention [7].…”

“…CTC-based approaches effectively tackle two common challenges found in non-autoregressive generation models: repetitive tokens and the need for a separate prediction of the target length [10]. CTC have been used in machine tranlation [8], [9], [11] and sentence summarization [5], [6].…”