Understanding credibility judgements for web search snippets

Abstract

Purpose It is well known that information behaviour can be biased in countless ways and that users of web search engines have difficulty in assessing the credibility of results. Yet, little is known about how search engine result page (SERP) listings are used to judge credibility and in which if any way such judgements are biased. The paper aims to discuss these issues. Design/methodology/approach Two studies are presented. The first collects data by means of a controlled, web-based user study (N=105). Studying judgements for three controversial topics, the paper examines the extent to which users agree on credibility, the extent to which judgements relate to those applied by objective assessors and to what extent judgements can be predicted by the users’ position on and prior knowledge of the topic. A second, qualitative study (N=9) utilises the same setup; however, transcribed think-aloud protocols provide an understanding of the cues participants use to estimate credibility. Findings The first study reveals that users are very uncertain when assessing credibility and their impressions often diverge from objective judges who have fact checked the sources. Little evidence is found indicating that judgements are biased by prior beliefs or knowledge, but differences are observed in the accuracy of judgements across topics. Qualitatively analysing think-aloud transcripts from participants think-aloud reveals ten categories of cues, which participants used to determine the credibility of results. Despite short listings, participants utilised diverse cues for the same listings. Even when the same cues were identified and utilised, different participants often interpreted these differently. Example transcripts show how participants reach varying conclusions, illustrate common mistakes made and highlight problems with existing SERP listings. Originality/value This study offers a novel perspective on how the credibility of SERP listings is interpreted when assessing search results. Especially striking is how the same short snippets provide diverse informational cues and how these cues can be interpreted differently depending on the user and his or her background. This finding is significant in terms of how search engine results should be presented and opens up the new challenge of discovering technological solutions, which allow users to better judge the credibility of information sources on the web.

Reliability Prediction for Health-Related Content: A Replicability Study

“…Other researches focused on determining how the search engine result page (SERP) listings are used to determine credibility through user studies [22]. More specifically, several studies have been conducted related to assessing the credibility of health-related content on the web.…”

Direct Answers in Google Search Results

“…Web search snippets in terms of quality evaluation are subject to credibility judgments. The same short snippets provide diverse informational cues and how these cues can be interpreted differently depending on the user and his or her background [25]. Snippets have different levels of readability and word complexity, however readability of snippets in Google and Bing search engines mismatched with the reading comprehension of children age 11 to 13 [26].…”

“…Varying relatively few words, and even their location, within a snippet can have a significant influence on the clickthrough of a snippet [31]. Normal snippets in previous works have mainly been evaluated as being informative enough for users of web search engines [25]. Recent research has been conducted on different age groups to see how normal snippets are perceived by younger and older search engine users [26].…”

Featured Snippets Results in Google Web Search: An Exploratory Study

“…For last few years it has been observed that some tests of changing length of regular snippets either on desktop or on mobile version have been done [4,5]. Research interest for regular snippets is about their function of being enough informative for users [6]. Recent research were conducted on different age groups to determine how regular snippets are perceived by younger or older search engine users [7].…”