Sequence Level Contrastive Learning for Text Summarization

Abstract

Contrastive learning models have achieved great success in unsupervised visual representation learning, which maximize the similarities between feature representations of different views of the same image, while minimize the similarities between feature representations of views of different images. In text summarization, the output summary is a shorter form of the input document and they have similar meanings. In this paper, we propose a contrastive learning model for supervised abstractive text summarization, where we view a document, its gold summary and its model generated summaries as different views of the same mean representation and maximize the similarities between them during training. We improve over a strong sequence-to-sequence text generation model (i.e., BART) on three different summarization datasets. Human evaluation also shows that our model achieves better faithfulness ratings compared to its counterpart without contrastive objectives. We release our code at https://github.com/xssstory/SeqCo.

BRIO: Bringing Order to Abstractive Summarization

“…However, they also exhibit similar problems of stability as reinforcement learning. Contrastive Learning Recently, contrastive learning (Hadsell et al, 2006) has been introduced into several conditional text generation tasks, such as machine translation Pan et al, 2021), text summarization (Cao and Wang, 2021;Xu et al, 2021;Sun and Li, 2021), and other tasks (Uehara et al, 2020;Cho et al, 2021;Lee et al, 2021b). Among these application scenarios, most work deployed contrastive learning in the latent representation space, following the framework proposed in .…”

“…GOLD (Pang and He, 2021) uses offline reinforcement learning to train the BART model by treating the reference summaries as the demonstrations, a different formulation that can also improve the performance of the original BART. SeqCo (Xu et al, 2021) and ConSum (Sun and Li, 2021) are two recent methods that aim to leverage contrastive learning to improve the performance of the abstractive summarization model (BART). Implementation Details In the following experiments, we use either BART or PEGASUS as a backbone.…”

CIAug: Equipping Interpolative Augmentation with Curriculum Learning

“…Analysing the convergence of CIAug For all benchmark datasets, we observe that CIAug reaches a benchmark F1 score faster than Mixup method, as shown in Figure 2. 2 As CIAug selects samples for Mixup based on a learning curriculum, it leads to generation of more suitable synthetic samples in a staggered manner resulting in better training (Xu et al, 2021) Using the NLPAug library (Ma, 2019) we substitute up to 10% of the words in each sentence with their synonyms found in WordNet (Feinerer and Hornik, 2020) and present the results in Table 4. We observe that both CIAug-NT and CIAug are more robust compared to regular Mixup by a difference of 6.72% and 6% respectively.…”

“…We propose CIAug 1 , a method which addresses these challenges by offering an augmentation procedure that selects samples in an adaptive fashion and is geometrically sound. CIAug's sampling strategy follows the idea that selecting easier mixing samples first and gradually increasing sample difficulty based on relative spatial position would generate more suitable synthetic inputs, resulting in better model training (Xu et al, 2021). This notion ties in with the framework of curriculum learning (Krueger and Dayan, 2009), where training data is presented in a similarly staggered way, increasing model capabilities (Bengio et al, 2009).…”

Jointly Learning Guidance Induction and Faithful Summary Generation via Conditional Variational Autoencoders

“…Contrastive learning methods encourage models to distinguish between positive and negative examples (Nan et al, 2021;Cao and Wang, 2021;Xu et al, 2022). Nan et al (2021) generate multiple summaries candidates by sampling from the pre-trained models and selecting positive and negative examples according to the question answer based metric.…”