A framework for evaluating automatic indexing or classification in the context of retrieval

Abstract

Tools for automatic subject assignment help deal with scale and sustainability in creating and enriching metadata, establishing more connections across and between resources and enhancing consistency. Although some software vendors and experimental researchers claim the tools can replace manual subject indexing, hard scientific evidence of their performance in operating information environments is scarce. A major reason for this is that research is usually conducted in laboratory conditions, excluding the complexities of real‐life systems and situations. The article reviews and discusses issues with existing evaluation approaches such as problems of aboutness and relevance assessments, implying the need to use more than a single “gold standard” method when evaluating indexing and retrieval, and proposes a comprehensive evaluation framework. The framework is informed by a systematic review of the literature on evaluation approaches: evaluating indexing quality directly through assessment by an evaluator or through comparison with a gold standard, evaluating the quality of computer‐assisted indexing directly in the context of an indexing workflow, and evaluating indexing quality indirectly through analyzing retrieval performance.

Potential and Challenges of Subject Access in Libraries Today on the Example of Swedish Libraries

“…Having reviewed a large number of automated indexing studies, Lancaster concluded that the research comparing automated versus manual indexing is "seriously flawed" (2003, p. 334). For a suggested comprehensive framework for evaluation of automatic indexing, see Golub et al (2016).…”

Some Thoughts on Preserving Functions of Library Catalogs in Networked Environments

“…Having reviewed a large number of automated indexing studies, Lancaster concluded that the research comparing automated versus manual indexing is problematic (, p. 334). For a suggested comprehensive framework for evaluation of automatic indexing see .…”

“…However, to what degree these three items may be combined and for what purposes, users and contexts needs to be investigated. A comprehensive methodology framework to evaluate these complex factors in the context of retrieval has been recently suggested .…”

O processo de construção do corpus de referência em Ciência da Informação

“…Assim, tais sistemas precisam ser avaliados na indexação de corpora de documentos cuja qualidade na indexação intelectual é a melhor possível, visando a correta avaliação e a comparação do desempenho dos mesmos. Tais corpora de documentos indexados são denominados corpus de referência (HASAN; NG, 2014) ou gold standard (Golub et al 2016), e permitem avaliar o quão próximo está a qualidade da indexação automática em relação à indexação intelectual (KIM et. al., 2013).…”