On choosing an effective automatic evaluation metric for microblog summarisation

Abstract

Popular microblogging services, such as Twitter, are engaging millions of users who constantly post and share information about news and current events each day, resulting in millions of messages discussing what is happening in the world. To help users obtain an overview of microblog content relating to topics and events that they are interested in, classical summarisation techniques from the newswire domain have been successfully applied and extended for use on microblogs. However, much of the current literature on microblog summarisation assumes that the summarisation evaluation measures that have been shown to be effective on newswire, are still appropriate for evaluating microblog summarisation. Hence, in this paper, we aim to determine whether the traditional automatic newswire summarisation evaluation metrics generalise to the task of microblog summarisation. In particular, using three microblog summarisation datasets, we determine a ranking of summarisation systems under three automatic summarisation evaluation metrics from the literature. We then compare and contrast this ranking of systems produced under each metric to system rankings produced through a qualitative user evaluation, with the aim of determining which metric best simulates human summarisation preferences. Our results indicate that, for the automatic evaluation metrics we investigate, they do not always concur with each other. Further, we find that Fraction of Topic Words better agrees with what users tell us about the quality and effectiveness of microblog summaries than the ROUGE-1 measure that is most commonly reported in the literature.

Estimating Summary Quality with Pairwise Preferences

“…Similarly, Giannakopoulos and Karkaletsis (2013) use machine learning to learn a linear combination of n-gram methods to evaluate summaries. Mackie et al (2014), Giannakopoulos (2013), and Cohan and Goharian (2016) investigate evaluation for microblog, multilingual, and scientific summarization, respectively. Our evaluation, on contrary, uses newswire datasets since this is the most prominent application domain for automatic summarization.…”

Experiments in Newswire Summarisation

“…In this section, we now examine if the ROUGE-2 metric aligns with user judgements, reproducing and validating previous findings -but generalising to the context of crowd-sourcing. This provides a measure of confidence in using crowdsourced evaluations of newswire summarisation, as has previously been demonstrated for microblog summarisation [11,12]. Our user study is conducted via CrowdFlower 1 , evaluating 5 baseline systems and 7 state-of-the-art systems, over the DUC 2004 dataset using summary texts from SumRepo.…”

Song Lyrics Summarization Inspired by Audio Thumbnailing

“…The key idea is that a summary should be similar to the original text in regard to characteristic criteria as the word distribution. (Mackie et al, 2014) find that topic words are a suitable metric to automatically evaluate micro blog summaries.…”