How users assess Web pages for information seeking

Abstract

In this article, we investigate the criteria used by online searchers when assessing the relevance of Web pages for information-seeking tasks. Twenty-four participants were given three tasks each, and they indicated the features of Web pages that they used when deciding about the usefulness of the pages in relation to the tasks. These tasks were presented within the context of a simulated work-task situation. We investigated the relative utility of features identified by participants (Web page content, structure, and quality) and how the importance of these features is affected by the type of information-seeking task performed and the stage of the search. The results of this study provide a set of criteria used by searchers to decide about the utility of Web pages for different types of tasks. Such criteria can have implications for the design of systems that use or recommend Web pages. IntroductionInformation retrieval (IR) systems aim to provide users with information that will help them in relation to the information need that they expressed to the system (typically in the form of a query). Searchers are then usually involved in the process of evaluating the utility (or the relevance) of the information (i.e., documents) that the IR system retrieves. One of the most common information-seeking situations entails the use of an Internet search engine (Jansen, Spink, & Saracevic, 2000). The availability of information on the World Wide Web (WWW) has established search engines as a major tool for IR and Web documents as a popular medium through which users access information.Assessing the utility of information in relation to an information need is a common task for online searchers. Studies on peoples' perceptions of the relevance of information demonstrate that a range of factors affect human judgements of relevance (e.g., Barry, 1994Barry, , 1998Cool, Belkin, & Kantor, 1993;Maglaughlin & Sonnenwald, 2002;Schamber, 1991). However, such studies often only consider formal textual documents such as journal and conference articles rather than the wide range of formally and informally produced multimedia documents found on the Web. The nature of the IR task on the WWW is different from that on more traditional IR systems (Jansen et al., 2000). One of the differences is the idiosyncrasy of the Web documents themselves. There is generally a large degree of variability in the quality, authority, and layout of Web pages. Moreover, the type of elements such pages contain (e.g., text, multimedia, links) can also vary to a large degree (Woodruff, Aoki, Brewer, Gauthier, & Rowe, 1996), creating a heterogeneous collection of documents distributed over distinct geographic areas.The motivation behind this study was to gain a better understanding of what features make a Web document useful for information seeking. We concentrated specifically on information-seeking tasks-finding Web pages that contain relevant or useful information-because this is one of the prominent uses of Web pages. It is also a task for which there exist ma...

Understanding an enriched multidimensional user relevance model by analyzing query logs

“…Therefore, judging the relevance of retrieved documents is a central task for search engine users. Indeed, relevance has been regarded as one of the fundamental and central concepts in information retrieval (Saracevic, 1975;Tombros et al, 2005), which influences the design and evaluation of IR models.…”

“…There have been numerous attempts (Tombros, Ruthven, & Jose, 2005;Xu & Yin, 2008;Xu & Chen, 2006;Zhang, Zhang, Lease, & Gwizdka, 2014) to understand users' search behaviors when retrieving information with search engines, foe example, relevance judgment, satisfaction or dissatisfaction with search results. Understanding how users conduct relevance judgment and what factors influence users' satisfaction with the search results would help researchers design more effective retrieval models and better evaluation methodologies, aiming to further improve users' search experience.…”

“…A large number of studies (Barry, 1998;Tombros et al, 2005;Xu & Yin, 2008;Xu & Chen, 2006;Zhang et al, 2014) have revealed that there exist a range of complex factors (e.g., topicality, novelty, reliability, understandability, and scope) affecting users' perception of relevance for the retrieved documents. However, the existing work is mainly based on small scale user studies, which may not reflect users' natural search scenarios, and the relevance judgments involved were made in a static way that cannot capture the dynamics of a user's information need, search interest, and habit.…”

Methods for Evaluating Interactive Information Retrieval Systems with Users

“…If the subject knows that all documents are from The New York Times, then the subject does not have to worry as much about source credibility. This suggests that subjects should be required to provide more and different kinds of relevance assessments when searching the web, than when searching closed corpora [271]. It is also the case that subjects may base their relevance judgments on more things than just the text (e.g., design, style, and images).…”

“…There have been a number of studies that have examined and studied (1) the concept of relevance (e.g., [33,197,224,232,235,236]); (2) the criteria users employ when making relevance assessments (e.g., [271]); and (3) techniques for measuring relevance (e.g., [84,162,272,283]). Suffice to say, the published research about how users make relevance assessments and the actual measures that researchers employ to collect relevance assessments are not very aligned.…”