Testing the stability of “wisdom of crowds” judgments of search results over time and their similarity with the search engine rankings

Abstract

Purpose: One of the under-explored aspects in the process of user information seeking behaviour is influence of time on relevance evaluation. It has been shown in previous studies that individual users might change their assessment of search results over time. It is also known that aggregated judgments of multiple individual users can lead to correct and reliable decisions; this phenomenon is known as the "wisdom of crowds". The aim of this study is to examine whether aggregated judgments will be more stable and thus more reliable over time than individual user judgments.Design/Methods: In this study two simple measures are proposed to calculate the aggregated judgments of search results and compare their reliability and stability to individual user judgments. In addition, the aggregated "wisdom of crowds" judgments were used as a means to compare the differences between human assessments of search results and search engine's rankings. A large-scale user study was conducted with 87 participants who evaluated two different queries and four diverse result sets twice, with an interval of two months. Two types of judgments were considered in this study: 1) relevance on a 4-point scale, and 2) ranking on a 10-point scale without ties.Findings: It was found that aggregated judgments are much more stable than individual user judgments, yet they are quite different from search engine rankings. Practical implications:The proposed "wisdom of crowds" based approach provides a reliable reference point for the evaluation of search engines. This is also important for exploring the need of personalization and adapting search engine's ranking over time to changes in users preferences. Originality/Value: This is a first study that applies the notion of "wisdom of crowds" to examine the under-explored phenomenon in the literature of "change in time" in user evaluation of relevance.Keywords: ranking, relevance judgment, change in time, wisdom of crowds Research paper IntroductionNumerous general models of information seeking and web searching behaviour have been proposed in the past such as (Ellis, 1989;Bates, 1989; Kuhlthau, 1991; Dervin, 1992; Johnson and Meishke, 1993;Marchionini, 1995; Spink, 1997; Wilson, 1999; Fisher et al., 2005;Knight and Spink, 2008; Du and Spink, 2010;Case, 2012). Relevance is a central notion in information science and is an important part of user information seeking models (Saracevic, 2007). This study investigates an under-explored topic in the literature (Saracevic, 2007): stability and change of user assessment of search results over time. Human evaluation of documents relevance is a complex process that requires coordination of multiple cognitive tasks (Du and Spink, 2011). User result evaluation is needed in many fields and has many purposes, hence it is important to understand the factors and phenomena behind it. This study aims to extend the understanding of the result evaluation component of the proposed web search behavior models, with respect to the temporal change factor. In this b...

Financial literacy resources in US public libraries: website analysis

“…Even search engines struggle to produce relevancy rankings that correspond to users' relevancy judgments, so this is a very difficult challenge to address, but it may be an aspect of libraries websites to consider improving, if possible (Zhitomirsky-Geffet et al, 2016). Relevancy rankings for search results would ideally take into consideration the currency of a result as well as how closely it aligns with search terms.…”

“…Most of the research in this area focuses on the relevancy of search engine results and has shown that individuals' assessments of relevancy tend to change over time. One common alternative to individual assessments of relevancy is to rely on the “wisdom of crowds” either through user studies or via click-through rates, which appears to improve the stability of relevancy judgments (Zhitomirsky-Geffet et al. , 2016; Zhitomirsky-Geffet et al ., 2018).…”

“…Most of the research in this area focuses on the relevancy of search engine results and has shown that individuals' assessments of relevancy tend to change over time. One common alternative to individual assessments of relevancy is to rely on the "wisdom of crowds" either through user studies or via click-through rates, which appears to improve the stability of relevancy judgments (Zhitomirsky-Geffet et al, 2016;Zhitomirsky-Geffet et al, 2018). Studies evaluating the information retrieval of library discovery tools have often relied on user observation in laboratory settings, log analysis or comparison of search results to student citations in particular projects or courses (Behnert and Lewandowski, 2017;Galbreath et al, 2021).…”

“Every Website Is a Puzzle!” : Facilitating Access to Common Website Features for People with Visual Impairments

“…If there's no predicted target page, users are redirected to the irst search result of a łfeeling-luckyž background search query constrained to the present domain. Users can mark pages as correct or incorrect for the intent, and such markings are considered later by the system in selecting the target page (e.g., select the page with the most correct marks), efectively leveraging the wisdom of the crowd [74]. If users think the target page is incorrect, they can request human help ś other users of the same website ś to ind and mark the page in real time, or later when someone is online.…”

Website removal from search engines due to copyright violation

“…Much research has focused on the issue of what strategies to undertake for sponsored results in order to achieve optimal effects (Sen, 2005; Cheng et al , 2018). Users prefer also results to be more adjusted to their expectations, or more customized, and eventually accept the changes to the displayed results (Zhitomirsky-Geffet et al , 2016). Users also tend use the search engine more when they see an interesting advertisement on the TV (Mukherjee and Jansen, 2017).…”