AnswerSumm: A Manually-Curated Dataset and Pipeline for Answer Summarization

Abstract

Community Question Answering (CQA) fora such as Stack Overflow and Yahoo! Answers contain a rich resource of answers to a wide range of community-based questions. Each question thread can receive a large number of answers with different perspectives. One goal of answer summarization is to produce a summary that reflects the range of answer perspectives. A major obstacle for this task is the absence of a dataset to provide supervision for producing such summaries. Recent works propose heuristics to create such data, but these are often noisy and do not cover all answer perspectives present. This work introduces a novel dataset of 4,631 CQA threads for answer summarization curated by professional linguists. Our pipeline gathers annotations for all subtasks of answer summarization, including relevant answer sentence selection, grouping these sentences based on perspectives, summarizing each perspective, and producing an overall summary. We analyze and benchmark state-of-the-art models on these subtasks and introduce a novel unsupervised approach for multi-perspective data augmentation that boosts summarization performance according to automatic evaluation. Finally, we propose reinforcement learning rewards to improve factual consistency and answer coverage and analyze areas for improvement.

mRedditSum: A Multimodal Abstractive Summarization Dataset of Reddit Threads with Images

“…However, our task aims to generate a unimodal output-that is, a purely textual summary. This is similar to the multimodal summarization done on the How2 Dataset (Sanabria et al, 2018 (Fabbri et al, 2022), ConvoSumm (Fabbri et al, 2021), SamSUM (Gliwa et al, 2019), CNN/DM (Nallapati et al, 2016), MSMO DailyMail (Zhu et al, 2018), and How2 (Yu et al, 2021, where a textual transcript of the video along with the video frames are generated into a text summary. (Yu et al, 2021) reported that incorporating the additional modality of the video frames into their summarization models showed improvement compared to text-only based models.…”

“…ConvoSumm (Fabbri et al, 2021) presented a dataset of 2000 summarized forum threads, 500 from each of 4 different domains including NYT articles, Reddit, StackExchange, and Email threads. Answer-Summ (Fabbri et al, 2022) is another dataset consisting of 4,631 question-answering discussion threads sourced from StackExchange. Answer-Summ shares the most similarities with our dataset, as they also summarize multi-speaker threads, and their annotation pipeline shares some similarities with ours.…”

“…We first cluster the comments to identify groups of comments that share a similar opinion. We follow the method described in AnswerSUMM (Fabbri et al, 2022) to allow for clusters of varying sizes and number. We use a RoBERTa-based model fine-tuned for semantic similarity to get sentence embeddings of the top-level comments from each thread.…”

“…With the increased popularity of online discussion forums like Reddit, discussion threads that consist of a post and various numbers of comments, have quickly accumulated. It thus becomes overwhelming for users to sift through the threads to find the information they seek, which in turn has led to the development of automated means for text-only discussion summarization (Bhatia et al, 2014;Fabbri et al, 2021Fabbri et al, , 2022.…”

Exploring Neural Models for Query-Focused Summarization

“…Recently, several query-focused summarization datasets have been introduced, which can be further divided into short-document datasets, whose source document length does not exceed the input limits of standard pretrained models, and longdocument datasets. Within short-document, queryfocused summarization, AnswerSumm (Fabbri et al, 2021c) is composed of summaries of answers to queries from StackExchange forums, while Wik-iHowQA (Deng et al, 2020) proposes the task of answer selection followed by the summarization of individual response articles to queries from the how-to site WikiHow. Within long-document summarization, WikiSum (Liu et al, 2018a) consists of Wikipedia article titles as queries, the first paragraph of the article as the summary, and documents referenced by the article as the input.…”