Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance

Abstract

Relevance is a, if not even the, key notion in information science in general and information retrieval in particular. This two-part critical review traces and synthesizes the scholarship on relevance over the past 30 years or so and provides an updated framework within which the still widely dissonant ideas and works about relevance might be interpreted and related. It is a continuation and update of a similar review that appeared in 1975 under the same title, considered here as being Part I. The present review is organized in two parts: Part II addresses the questions related to nature and manifestations of relevance, and Part III addresses questions related to relevance behavior and effects. In Part II, the nature of relevance is discussed in terms of meaning ascribed to relevance, theories used or proposed, and models that have been developed. The manifestations of relevance are classified as to several kinds of relevance that form an interdependent system of relevancies. In Part III, relevance behavior and effects are synthesized using experimental and observational works that incorporated data. In both parts, each section concludes with a summary that in effect provides an interpretation and synthesis of contemporary thinking on the topic treated or suggests hypotheses for future research. Analyses of some of the major trends that shape relevance work are offered in conclusions. Prologue to Part III: How It Is Connected and What This Work Is All AboutTo provide a continuation from the preceding Part II, a few basic descriptions about this work are repeated.As stated in the Preface to Part II, in 1975 I published a review about relevance under the same title, without, of course, "Part I" in the title (Saracevic, 1975). There was no plan then to have another related review 30 years later-but things happen. The intent of the 1975 work was "to explore the meaning of relevance as it has evolved in information science and to provide a framework within which various interpretations of relevance can be related" (Saracevic, 1975, p. 321).Building on the examination of relevance in the preceding (1975) review, this work (2007) follows the travails of relevance in information science for the past 30 years. It is an update. The aim of this work is still substantially the same: It is an attempt to trace the evolution of thinking on relevance in information science for the past three decades and to provide an updated, contemporary framework within which the still widely dissonant ideas on relevance might be interpreted and related to one another.The organization of the present review, offered in two parts, addresses the questions related to relevance nature, manifestations, behavior, and effects. Following the Introduction and a Historical Footnote, the preceding Part II (this issue, pp. 1915-1933) started with a general section on the nature of relevance by synthesizing its meanings, following with sections on theories and models of relevance that are, in effect, further elaborations on the nature of relevance. Part ...

Exploratory computing: a comprehensive approach to data sensemaking

“…A key notion in the field of information retrieval but also in information science in general is that of relevance [9] whose definition has stimulated widely dissonant ideas and works. As introduced in Sect.…”

Testing the stability of “wisdom of crowds” judgments of search results over time and their similarity with the search engine rankings

“…While, in general, the order of results might have influenced the judgments (Bar-Ilan et al, 2009;Joachims et al, 2007), it has been shown that when the number of results is small than this influence is insignificant (Saracevic, 2007).…”

“…As stated by (Saracevic, 2007(Saracevic, , p. 2139: "The role of research is to make relevance complexity more comprehensible formally and possibly even more predictable". Accurate ranking of search results according to the users' preferences is one of the most important challenges of the modern search systems.…”

“…This study aims to extend the understanding of the result evaluation component of the proposed web search behavior models, with respect to the temporal change factor. In this broad context, this research contributes to modelling the change in user relevance evaluation behaviour over time.As stated by (Saracevic, 2007(Saracevic, , p. 2139: "The role of research is to make relevance complexity more comprehensible formally and possibly even more predictable". Accurate ranking of search results according to the users' preferences is one of the most important challenges of the modern search systems.…”

Relevance behaviour in TREC

“…Assessing a document as relevant to an information need can be a complex decision that may be affected by the searcher's knowledge, the task being conducted, the collection being searched, the retrieval results and properties of the document itself (Barry and Schamber, 1998;Harter, 1996;Ruthven, 2005;Saracevic, 2007a;Saracevic, 2007b;Spink et al, 1998;Vakkari and Hakala, 2000;Voorhees, 2001). Analysing relevance assessments can help us understand how people make judgments about relevance, how people solve information problems and, through collection of assessments within test collections, help us evaluate the performance of retrieval systems.…”