A Study of Snippet Length and Informativeness

Abstract

e design and presentation of a Search Engine Results Page (SERP) has been subject to much research. With many contemporary aspects of the SERP now under scrutiny, work still remains in investigating more traditional SERP components, such as the result summary. Prior studies have examined a variety of di erent aspects of result summaries, but in this paper we investigate the in uence of result summary length on search behaviour, performance and user experience. To this end, we designed and conducted a withinsubjects experiment using the TREC AQUAINT news collection with 53 participants. Using Kullback-Leibler distance as a measure of information gain, we examined result summaries of di erent lengths and selected four conditions where the change in information gain was the greatest: (i) title only; (ii) title plus one snippet; (iii) title plus two snippets; and (iv) title plus four snippets. Findings show that participants broadly preferred longer result summaries, as they were perceived to be more informative. However, their performance in terms of correctly identifying relevant documents was similar across all four conditions. Furthermore, while the participants felt that longer summaries were more informative, empirical observations suggest otherwise; while participants were more likely to click on relevant items given longer summaries, they also were more likely to click on non-relevant items. is shows that longer is not necessarily be er, though participants perceived that to be the case -and second, they reveal a positive relationship between the length and informativeness of summaries and their a ractiveness (i.e. clickthrough rates). ese ndings show that there are tensions between perception and performance when designing result summaries that need to be taken into account.

Everything you always wanted to know about a dataset: Studies in data summarisation

“…By comparison, the crowd needed on average 63 words for the same datasets in Set − 5. It would be interesting to explore how the length of the summaries impacts on their perceived usefulness by readers or on their potential information gain (Maxwell et al, 2017), also in the context of the summary template we propose in Section 6.…”

Information Scent, Searching and Stopping

“…Log interaction data was obtained from a within-subjects user study by Maxwell et al [21], using the same collection and retrieval model as above. In the study, 53 subjects undertook ad-hoc topic retrieval using the same configuration of search engine and corpus as described above.…”

“…These strategies are grounded using analysis from a prior user study [21] examining information scent. We report on a large-scale simulation, allowing us to address our two main research questions.…”

“…[8,21,27,33,37]). Wu et al [37] conducted a user study where the scent of the first SERP was manipulated.…”

“…A study by Ong et al [27] replicated the same experimental setup as used by Wu et al [37], but for both desktop and mobile environments, where similar findings were observed -subjects using the desktop interface however tended to perform better. Maxwell et al [21] conducted a user study where information scent was varied by manipulating the length of snippets (changing proximal cues) as opposed to manipulating performance of SERPs as done before [27,37]. It was found that as result snippets increased in length from title only to title plus four line summaries, subjects examined fewer snippets -and were more likely to click on documents, but with lower accuracy.…”