Relevance criteria identified by health information users during Web searches

Abstract

This article focuses on the relevance judgments made by health information users who use the Web. Health information users were conceptualized as motivated information users concerned about how an environmental issue affects their health. Users identified their own environmental health interests and conducted a Web search of a particular environmental health Web site. Users were asked to identify (by highlighting with a mouse) the criteria they use to assess relevance in both Web search engine surrogates and full-text Web documents. Content analysis of document criteria highlighted by users identified the criteria these users relied on most often. Key criteria identified included (in order of frequency of appearance) research, topic, scope, data, influence, affiliation, Web characteristics, and authority/ person. A power-law distribution of criteria was observed (a few criteria represented most of the highlighted regions, with a long tail of occasionally used criteria). Implications of this work are that information retrieval (IR) systems should be tailored in terms of users' tendencies to rely on certain document criteria, and that relevance research should combine methods to gather richer, contextualized data. Metadata for IR systems, such as that used in search engine surrogates, could be improved by taking into account actual usage of relevance criteria. Such metadata should be user-centered (based on data from users, as in this study) and contextappropriate (fit to users' situations and tasks).

Contextual Metadata in Digital Aggregations: Application of Collection-Level Subject Metadata and Its Role in User Interactions and Information Retrieval

“…As demonstrated by previous research (e.g., Crystal & Greenberg, 2006;Drori, 2003;Wang & Soergel, 1998), users benefit from search result interfaces that display subject metadata and perceive the free-text abstract and subject keywords to be among the most useful metadata elements to judge the relevance of retrieved documents. However, these studies examined user interactions with item-level metadata describing individual information objects in IR systems and search engines.…”

Design factors affecting relevance judgment behaviour in the context of metadata surrogates

“…Other criteria used for relevance judgment may include the currency, quality, authority and availability of the resource as well as users' subject background and characteristics. Lists of such criteria have been proposed by several researchers (see for example, Barry & Schamber, 1998;Wang & Soergel, 1998;Crystal & Greenberg, 2006). Researchers in this field have also argued that the importance of relevance criteria may change throughout the information seeking and searching process exposing its dynamic dimension (Rieh, 2002;Tang & Solomon, 2001;Crystal & Greenberg, 2006).…”

“…Lists of such criteria have been proposed by several researchers (see for example, Barry & Schamber, 1998;Wang & Soergel, 1998;Crystal & Greenberg, 2006). Researchers in this field have also argued that the importance of relevance criteria may change throughout the information seeking and searching process exposing its dynamic dimension (Rieh, 2002;Tang & Solomon, 2001;Crystal & Greenberg, 2006). This change can be attributed to several individual characteristics, such as topic familiarity, level of interest with the topic and confidence in assessing relevance (Ruthven, et al, 2008).…”

HealthFinland—A national semantic publishing network and portal for health information

“…The presentation of search results was designed to show rich metadata that helps users quickly find relevant items and filter out nonrelevant results. The design of the search results listing was based on research by Crystal and Greenberg that evaluated the usefulness of metadata for health information seekers [3]. The display of facets and search results is shown in Figure 5.…”