Query-based summarization of discussion threads

Abstract

In this paper, we address query-based summarization of discussion threads. New users can profit from the information shared in the forum, Please check if the inserted city and country names in the affiliations are correct. if they can find back the previously posted information. However, discussion threads on a single topic can easily comprise dozens or hundreds of individual posts. Our aim is to summarize forum threads given real web search queries. We created a data set with search queries from a discussion forum’s search engine log and the discussion threads that were clicked by the user who entered the query. For 120 thread–query combinations, a reference summary was made by five different human raters. We compared two methods for automatic summarization of the threads: a query-independent method based on post features, and Maximum Marginal Relevance (MMR), a method that takes the query into account. We also compared four different word embeddings representations as alternative for standard word vectors in extractive summarization. We find (1) that the agreement between human summarizers does not improve when a query is provided that: (2) the query-independent post features as well as a centroid-based baseline outperform MMR by a large margin; (3) combining the post features with query similarity gives a small improvement over the use of post features alone; and (4) for the word embeddings, a match in domain appears to be more important than corpus size and dimensionality. However, the differences between the models were not reflected by differences in quality of the summaries created with help of these models. We conclude that query-based summarization with web queries is challenging because the queries are short, and a click on a result is not a direct indicator for the relevance of the result.

A Survey of Evaluation Metrics Used for NLG Systems

“…In general, it is desired to have a high inter-annotator agreement (IAA), which is usually measured using Cohen's Kappa or Fleiss Kappa co-efficient or KrippendorffâĂŹs alpha. Alternatively, although not popularly, IAA could be measured using Jaccard similarity, or an F1-measure (based on precision and recall between annotators) [160]. Achieving a high-enough inter-annotator agreement is more difficult on some NLG tasks which have room for subjectivity [2].…”

Towards Context-Aware Opinion Summarization for Monitoring Social Impact of News

“…A word embedding is a learned representation for text where words that have the same meaning have a similar representation. This kind of representation has been successful in extractive summarization [32]. WordNet [11] is the most commonly used technique for capturing and processing the semantic meaning of terms; however, it has not been so much when summarizing opinions.…”

Automatic summarisation: 25 years On

“…The move from single-document summarisation to multi-document summarisation was largely promoted by the need to deal with the increasing amount of information available online. Initially, the methods were applied to newswire texts, but more recent work focused on summarisation of user-generated content such as discussions on forums (Tigelaar, Op Den Akker, and Hiemstra, 2010;Verberne, Krahmer, Wubben, and van den Bosch, 2019), customer reviews with the output focused on product features to assist customers' decision making (Feiguina and Lapalme, 2007).…”

“…Initially, the methods were applied to newswire texts, but more recent work focused on summarisation of user-generated content such as discussions on forums (Tigelaar, Op Den Akker, and Hiemstra 2010; Verberne et al . 2019) and customer reviews with the output focused on product features to assist customers’ decision making (Feiguina and Lapalme 2007).…”