Search effort degrades search output but improves task outcome

Abstract

We analyzed how effort in searching is associated with search output and task outcome. In a field study, we examined how students' search effort for an assigned learning task was associated with precision and relative recall, and how this was associated to the quality of learning outcome. The study subjects were 41 medical students writing essays for a class in medicine. Searching in Medline was part of their assignment. The data comprised students' search logs in Medline, their assessment of the usefulness of references retrieved, a questionnaire concerning the search process, and evaluation scores of the essays given by the teachers. Pearson correlation was calculated for answering the research questions. Finally, a path model for predicting task outcome was built. We found that effort in the search process degraded precision but improved task outcome. There were two major mechanisms reducing precision while enhancing task outcome. Effort in expanding Medical Subject Heading (MeSH) terms within search sessions and effort in assessing and exploring documents in the result list between the sessions degraded precision, but led to better task outcome. Thus, human effort compensated bad retrieval results on the way to good task outcome. Findings suggest that traditional effectiveness measures in information retrieval should be complemented with evaluation measures for search process and outcome.

Searching as learning: A systematization based on literature

“…The more users were able to focus on results inspection instead of querying and on working with information in documents found instead of result inspection, the better this seemed to be to the process of construction for proceeding in the task [cf. 30,39,42]. Also a recent query log study [47] seems to hint at this.…”

“…Although it has produced some interesting empirical results [30,39,49], it seems to lack a coherent conceptual framework on which to build empirical research. One of the major conceptualizations in task-based studies has been Kuhlthau's ISP model, which is deeply rooted in pedagogy [1].…”

“…Interacting with sources is the most crucial phase for leaning [17,30]. Users familiarize themselves with the sources selected first with the aim of revising their understanding concerning the topic.…”

“…Also studies by Bron and colleagues [41] and by Vakkari and Huuskonen [30] analysed how search process variables contributed to the outcome of the search. The outcome was in the former the quality of the research question created with the help of material retrieved, and in the latter the proportion of useful retrieved documents cited in the assigned essay and the quality of the essay.…”

“…Also the type of information needed and used changes during the information search process in task performance [29,32]. An increasing share of consulted sources is used in the outcome [28,30]. Table 2 suggests how changes in the various stages of searching can be used as criteria of learning during the search process.…”