Relevance judgment: What do information users consider beyond topicality?

Abstract

How does an information user perceive a document as relevant?The literature on relevance has identified numerous factors affecting such a judgment. Taking a cognitive approach, this study focuses on the criteria users employ in making relevance judgment beyond topicality. On the basis of Grice's theory of communication, we propose a five-factor model of relevance: topicality, novelty, reliability, understandability, and scope. Data are collected from a semicontrolled survey and analyzed by following a psychometric procedure. Topicality and novelty are found to be the two essential relevance criteria. Understandability and reliability are also found to be significant, but scope is not. The theoretical and practical implications of this study are discussed.confirmatory studies adopt a positivist perspective and employ a statistical hypothesis-testing procedure, which helps further the test of the validity of identified factors and weed out the insignificant ones (Tashakkori & Teddlie, 1998). Unfortunately, almost no study of relevance judgment has adopted a confirmatory approach. Consequently, the importance of many relevance criteria is still unclear.With a focus on users' relevance judgment, the purposes of this study are (1) to identify a set of core relevance criteria using a theory-driven approach and (2) to test the validity of these factors with a rigorous psychometric approach. The rest of the article is organized as follows: We review the literature on relevance and relevance judgment next. After that, we identify a set of core factors based on Grice's (1989) communication theory, which leads us to our research model and hypotheses. The empirical study is then discussed and the data analysis reported. Finally, we discuss the theoretical and empirical implications of our findings.

Understanding an enriched multidimensional user relevance model by analyzing query logs

“…First, relevance is multidimensional, that is, users tend to adopt multiple criteria or factors beyond topicality when performing their relevance judgments (Barry, 1998;Xu & Chen, 2006;Zhang et al, 2014). Second, relevance is a dynamic phenomenon in a sense that a user may judge the same document as relevant at a certain point of time but irrelevant at another point (Tiamiyu & Ajiferuke, 1988;Katzer & Snyder, 1990).…”

“…In this article, we focus on the MURM that is most related to ours. Xu and Chen (2006) studied the multicriteria of users' relevance judgment (such as topicality, novelty, reliability, understandability, and scope) based on the semicontrolled user survey data, and drew a conclusion that the topicality and novelty are the essential relevance criteria. Xu and Chen's study focused on the psychological/subjective state of users when answering questionnaires, whereas ours focuses more on the interaction behaviors of users, for example, clicking and dwelling on webpages in a real-world search environment and obtain different findings.…”

“…There have been numerous attempts (Tombros, Ruthven, & Jose, 2005;Xu & Yin, 2008;Xu & Chen, 2006;Zhang, Zhang, Lease, & Gwizdka, 2014) to understand users' search behaviors when retrieving information with search engines, foe example, relevance judgment, satisfaction or dissatisfaction with search results. Understanding how users conduct relevance judgment and what factors influence users' satisfaction with the search results would help researchers design more effective retrieval models and better evaluation methodologies, aiming to further improve users' search experience.…”

“…with the search engine, as an important supplement to the numerous existing work based on user studies. Specifically, we analyze how different factors affect the users' perception of relevance on retrieved documents within the framework of the Multidimensional User Relevance Model (MURM; Xu & Chen, 2006;Zhang et al, 2014). The existing MURM model includes five dimensions: topicality, novelty, reliability, understandability, and scope.…”

“…A large number of studies (Barry, 1998;Tombros et al, 2005;Xu & Yin, 2008;Xu & Chen, 2006;Zhang et al, 2014) have revealed that there exist a range of complex factors (e.g., topicality, novelty, reliability, understandability, and scope) affecting users' perception of relevance for the retrieved documents. However, the existing work is mainly based on small scale user studies, which may not reflect users' natural search scenarios, and the relevance judgments involved were made in a static way that cannot capture the dynamics of a user's information need, search interest, and habit.…”