CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization

Abstract

We study generating abstractive summaries that are faithful and factually consistent with the given articles. A novel contrastive learning formulation is presented, which leverages both reference summaries, as positive training data, and automatically generated erroneous summaries, as negative training data, to train summarization systems that are better at distinguishing between them. We further design four types of strategies for creating negative samples, to resemble errors made commonly by two state-of-the-art models, BART and PEGASUS, found in our new human annotations of summary errors. Experiments on XSum and CNN/Daily Mail show that our contrastive learning framework is robust across datasets and models. It consistently produces more factual summaries than strong comparisons with post error correction, entailmentbased reranking, and unlikelihood training, according to QA-based factuality evaluation. Human judges echo the observation and find that our model summaries correct more errors.

Learning Discriminative Representations for Open Relation Extraction with Instance Ranking and Label Calibration

“…Entity Replacing for Hard Negative: We choose T5 (Raffel et al, 2019) to generate the most similar word to head or tail entity, and then replace the head or tail entity with its augmented word to obtain the hard negative instance, which possesses the similar entity and context to original instance. Entity Swap for Semi-Hard Negative: To con-struct a semi-hard negative instance for the original instance, we follow the setting of Entity Swap (Cao and Wang, 2021), which swaps the target entities with other randomly selected entities of the same entity type in the original instance.…”

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorial Abstracts

“…We will introduce effective approaches that use alternative supervision sources for IE, that is, to use supervision signals from related tasks to make up for the lack of quantity and comprehensiveness in IE-specific training data. This includes indirect supervision sources such as question answering and reading comprehension Lyu et al, 2021;Levy et al, 2017;Li et al, 2019b;Du and Cardie, 2020), natural language inference (Li et al, 2022a;Yin et al, 2020) and generation (Lu et al, 2021;. We will also cover the use of weak supervision sources such as structural texts (e.g., Wikipedia) (Ji et al, 2017;Zhou et al, 2018) and global biases (Ning et al, 2018b).…”

“…Shu et al (2021) propose to improve logicto-text generation models by designing rule-based data augmentation to create contrastive examples to cover variations of logic forms paired with diverse natural language expressions to improve the generalizability. CLIFF (Cao and Wang, 2021) propose to improve faithful and factual consistency for abstractive summarization by contrasting reference summaries as positive training data and automatically generated erroneous summaries as negative training data. Wu et al (2020a) also propose to use contrastive learning for unsupervised referencefree summary quality evaluation.…”

“…We plan cover the following: Contrastive Data Augmentation for NLP (Shen et al, 2020;; Text Classification (Fang et al, 2020;Kachuee et al, 2020;Suresh and Ong, 2021;Du et al, 2021;Carlsson et al, 2021;Qiu et al, 2021;Klein and Nabi, 2021); Sentence Embeddings Sedghamiz et al, 2021) including Quick-Thought (Logeswaran and Lee, 2018),Sentence-BERT (Reimers and Gurevych, 2019), Info-Sentence BERT , SimCSE (Gao et al, 2021b), DeCLUTR (Giorgi et al, 2020, ConSERT (Yan et al, 2021b), Di-alogueCSE (Liu et al, 2021a). We will also cover discourse analysis (Iter et al, 2020;Kiyomaru and Kurohashi, 2021); Information Extraction (Qin et al, 2020; Machine Translation Vamvas and Sennrich, 2021); Question Answering (Karpukhin et al, 2020;You et al, 2021;; Summarization (Duan et al, 2019; including faithfulness (Cao and Wang, 2021), summary evaluation (Wu et al, 2020a), multilingual summarization , and dialogue summarization ; Text Generation (Chai et al, 2021; including logicconsistent text generation (Shu et al, 2021), paraphrase generation , grammatical error correction , dialogue generation (Cai et al, 2020), x-ray report generation Yan et al, 2021a), data-to-text generation (Uehara et al, 2020); Few-shot Learning Wang et al, 2021c;Luo et al, 2021;Das et al, 2021); Language Model Contrastive Pretraining (Wu…”

Text Summarization Method Based on Gated Attention Graph Neural Network

“…To improve the robustness of the graph structure, refs. [ 26 , 27 , 28 , 29 , 30 ] introduced contrast learning in the model for pulling positive samples close to push away negative samples. Most of the encoders of existing models are based on sentences or words with low reliability, and the joint summary model proposed in [ 31 ] improves the performance of summary generation.…”