Term and citation retrieval: A field study

Abstract

Abstract-Therelative efficacy of searching by terms and by citations is investigated with real searches collected in health sciences libraries. The objective is to seek evidence to confirm or refute findings from a controlled pilot study, and to understand the factors at work in operational search environments.Overall confirmation was found. In both the pilot and field studies, the improvement of the odds that overlap items retrieved would be relevant or partially relevant was truly astounding. If an item was retrieved from both MEDLINE@ and SCISEARCHB, it was six times more likely that it would be relevant or partially relevant as opposed to being not relevant, and 8.4 times more likely for definitely relevant retrievals. In the field setting, citation searching was able to add an average of 24% recall to traditional subject retrieval. Term or citation searching from the open literature produced lower precision results. Attempts to identify distinguishing characteristics in queries which might benefit most from additional citation searches proved to be inconclusive.In spite of the obvious gain shown by citation searching, online access of citation databases has been hampered by their relative high cost.The reported work is an extension of a pilot study of the characteristics and retrieval effectiveness of two subject searching modes. The two search approaches available on commercial bibliographic databases are semantic retrieval based on text words, assigned keywords and descriptors, and pragmatic retrieval based on citations. The earlier study was an experiment performed on a data file with narrow subject focus (Pao & Worthen, 1989). The database was constructed such that its documents content was retrievable by descriptors and text words as well as by cited references. Thus, direct comparison of the retrieval results was possible from parallel searches using appropriate terms and citations on identical queries.Citation searching in the control setting was found to add an average of 14% of relevant documents to a search. This confirms earlier indications from other researchers that citation searching complements searching by terms (McCain, 1989;Pao, 1986;Salton, 1971). A logical follow-up question is whether these results could be of practical use to the online searcher. Can the searcher expect similar results when searches are done on commercially available databases? For example, what is the percentage of MEDLINE@ search topics which could benefit from a citation search? One also wonders how generalizable and how stable the findings are if a sample of real searches was collected from libraries where a wide variety of topics was searched. While only a few overlap items were found, these common documents tended to be highly relevant to the search topic. What are the odds that retrieved items derived from both types of search methods are relevant? Is a higher yield from a citation search related to specific types of topics? Obviously, knowledge of this type could be useful to the online searcher.This second stu...

Author‐rated importance of cited references in biology and psychology publications

“…With nine topic searches, McCain found only 10 percent of overlap between descriptor-based and citation-based retrievals. Pao's (1993) data yielded an even lower percentage of overlapping, 4.8 percent from a total of 5,836 unique items. Researchers thus strongly recommend conducting composite/complementary searches on both subject-specific databases and citation databases (McCain, 1989;Pao, 1993;Brown, 1998).…”

The concept of relevance in IR

“…Eisenberg and Hu (1987) used a seven-point category scale. Three-category rating scales of relevance have been used by, for example, Saracevic (1969), Lancaster (1969), Saracevic and Su (1989), Pao (1993), Borlund and Ingwersen (1997), and Borlund (2000a). As pointed out by Tang et al (1999, p. 256), it is not the number of categories alone that describe a rating scale because the scales may differ on how relevance categories are defined and on the style of category anchoring.…”

“…The three categories of relevance applied by, for example, Saracevic (1969), are labeled relevant, partially relevant, and nonrelevant. Most IR experiments (e.g., Borlund, 2000a;Borlund & Ingwersen, 1997;Pao, 1993;Saracevic, 1969;Sacarevic & Su, 1989) make use of three scaled relevance categories. The initial reason for that might have been an intuitive understanding of the need for more than two categories in user-based relevance assessment in IR.…”

Relevance: An Interdisciplinary and Information Science Perspective

“…They indicate "partially relevant" as any document considered only somewhat, or in some part, related to the question or to any part of the question. Pao (1993) and Su (1994) also used a relevant, partially relevant, and not relevant methodology. In most cases, however, even though users and/or judges were able to conceptualize a meaningful difference between relevant and partially relevant documents, the experimental results were combined, collapsed, or grouped into a single category of "relevant" for the purpose of analysis.…”