A Method for Comparing Large Scale Inter-indexer Consistency Using IR Modeling

Wolfram, Dietmar; Olson, Hope A.

doi:10.29173/cais250

Cited by 6 publications

(11 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The three‐dimensional space may be extended to n dimensions when n different index terms are present. Wolfram and Olson () applied the concept of document space in the VSM to the terms assigned by a group of indexers to a document, and defined an indexer/tagger Space. Wolfram and Olson proposed a method, the inter‐indexer consistency density (ICD), by calculating the distance between each indexer/tagger's resulting vector and the indexing centroid (or average vector across all indexers/taggers).…”

Section: Assessment Of Tagging Quality: Methodsmentioning

confidence: 99%

“…Wolfram and Olson ()'s inter‐indexer (tagger) consistency density (ICD) is calculated as follows:

I C D = \frac{\sum_{i = 1}^{m} s i m (I_{i} ​, C)}{m}

…”

Section: Assessment Of Tagging Quality: Methodsmentioning

confidence: 99%

“…Researchers have characterized social tagging as user indexing and discussed user‐based, uncontrolled indexing (Choi, 2010a–c, 2013; Rafferty & Hidderley, ; Smith, ; Svarre & Lykke, ; Wolfram & Olson, ; Wolfram et al, ). Smith () asserted that tagging is better than subject headings by investigating tags assigned in LibraryThing and the subject headings assigned by the Library of Congress Subject Headings (LCSH).…”

Section: Literature Reviewmentioning

confidence: 99%

“…Rafferty and Hidderley () examined dialogic indexing practice by analyzing tags in Flickr. Wolfram and Olson () and Wolfram et al () analyzed the inter‐indexer consistency of tagging regarding several subject areas by employing the vector space model. Choi (2010a) focused on bridging the gap of insufficiency of studies on vocabulary analysis by comparing user‐generated tags with professionally‐generated index terms regarding web resources.…”

Section: Literature Reviewmentioning

confidence: 99%

“…This research addresses the following research gaps: (a) no indexing consistency research measuring consistency on web resources across professionally indexed web directories, for example, subject gateways; (b) lack of extensive indexing consistency research covering a variety of kinds of web resources. Although there have been indexing consistency studies (Kipp, ; Wolfram & Olson, ; Wolfram et al, ), with a large number of indexers or taggers, they examined only a limited scope of documents, for example, journal articles on CiteULike.org.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

A complete assessment of tagging quality: A consolidated methodology

Choi

2014

Asso for Info Science & Tech

View full text Add to dashboard Cite

This paper presents a methodological discussion of a study of tagging quality in subject indexing. The data analysis in the study was divided into 3 phases: analysis of indexing consistency, analysis of tagging effectiveness, and analysis of the semantic values of tags. To analyze indexing consistency, this study employed the vector space model-based indexing consistency measures. An analysis of tagging effectiveness with tagging exhaustivity and tag specificity was conducted to ameliorate the drawbacks of consistency analysis based on only the quantitative measures of vocabulary matching. To further investigate the semantic values of tags at various levels of specificity, a latent semantic analysis (LSA) was conducted. To test statistical significance for the relation between tag specificity and semantic quality, correlation analysis was conducted. This research demonstrates the potential of tags for web document indexing with a complete assessment of tagging quality and provides a basis for further study of the strengths and limitations of tagging.

show abstract

Section: Assessment Of Tagging Quality: Methodsmentioning

confidence: 99%

“…Wolfram and Olson ()'s inter‐indexer (tagger) consistency density (ICD) is calculated as follows:

I C D = \frac{\sum_{i = 1}^{m} s i m (I_{i} ​, C)}{m}

…”

Section: Assessment Of Tagging Quality: Methodsmentioning

confidence: 99%

Section: Literature Reviewmentioning

confidence: 99%

Section: Literature Reviewmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A complete assessment of tagging quality: A consolidated methodology

Choi

2014

Asso for Info Science & Tech

View full text Add to dashboard Cite

show abstract

Inter‐indexer consistency in graphic materials indexing at the National Library of Wales

Hughes

Rafferty

2011

Journal of Documentation

View full text Add to dashboard Cite

Purpose -This paper seeks to report a project to investigate the degree of inter-indexer consistency in the assignment of controlled vocabulary topical subject index terms to identical graphical images by different indexers at the National Library of Wales (NLW). Design/methodology/approach -An experimental quantitative methodology was devised to investigate inter-indexer consistency. Additionally, the project investigated the relationship, if any, between indexing exhaustivity and consistency, and the relationship, if any, between indexing consistency/exhaustivity and broad category of graphic format. Findings -Inter-indexer consistency in the assignment of topical subject index terms to graphic materials at the NLW was found to be generally low and highly variable. Inter-indexer consistency fell within the range 10.8 per cent to 48.0 per cent. Indexing exhaustivity varied substantially from indexer to indexer, with a mean assignment of 3.8 terms by each indexer to each image, falling within the range 2.5 to 4.7 terms. The broad category of graphic format, whether photographic or non-photographic, was found to have little influence on either inter-indexer consistency or indexing exhaustivity. Indexing exhaustivity and inter-indexer consistency exhibited a tendency toward a direct, positive relationship. The findings are necessarily limited as this is a small-scale study within a single institution. Originality/value -Previous consistency studies have almost exclusively investigated the indexing of print materials, with very little research published for non-print media. With the literature also rich in discussion of the added complexities of subjectively representing the intellectual content of visual media, this study attempts to enrich existing knowledge on indexing consistency for graphic materials and to address a noticeable gap in information theory. IntroductionInter-indexer consistency is a quantitative measure of the degree of agreement between different indexers indexing the same document within an information retrieval system. Although the literature shows only a tentative correlation between indexing consistency and retrieval effectiveness, many see consistency as a benchmark of quality both in service provision to information seekers and potent resource discovery. Previous consistency studies are abundant yet have almost exclusively investigated the indexing of print materials, with very little research published for non-print media. With the literature also rich in discussion of the added complexities of representing the intellectual content of visual media, this study focuses on indexing consistency for graphic materials in an attempt to address a noticeable gap in information theory.

show abstract

Obsolescence in subject description

Buckland

2012

Journal of Documentation

View full text Add to dashboard Cite

Purpose -The paper aims to explain the character and causes of obsolescence in assigned subject descriptors. Design/methodology/approach -The paper takes the form of a conceptual analysis with examples and reference to existing literature. Findings -Subject description comes in two forms: assigning the name or code of a subject to a document and assigning a document to a named subject category. Each method associates a document with the name of a subject. This naming activity is the site of tensions between the procedural need of information systems for stable records and the inherent multiplicity and instability of linguistic expressions. As languages change, previously assigned subject descriptions become obsolescent. The issues, tensions, and compromises involved are introduced. Originality/value -Drawing on the work of Robert Fairthorne and others, an explanation of the unavoidable obsolescence of assigned subject headings is presented. The discussion relates to libraries, but the same issues arise in any context in which subject description is expected to remain useful for an extended period of time.

show abstract

A Method for Comparing Large Scale Inter-indexer Consistency Using IR Modeling

Cited by 6 publications

References 15 publications

A complete assessment of tagging quality: A consolidated methodology

A complete assessment of tagging quality: A consolidated methodology

Inter‐indexer consistency in graphic materials indexing at the National Library of Wales

Obsolescence in subject description

Contact Info

Product

Resources

About