Automatic evaluation of metadata quality in digital repositories

Abstract

Due to recent developments in automatic metadata generation and interoperability between digital repositories, the production of metadata is now vastly surpassing manual quality control capabilities. Abandoning quality control altogether is problematic, because low quality metadata compromise the effectiveness of services that repositories provide to their users. To address this problem, we present a set of scalable quality metrics for metadata based on the Bruce & Hillman framework for metadata quality control. We perform three experiments to evaluate our metrics: 1) the degree of correlation between the metrics and manual quality reviews, 2) the discriminatory power between metadata sets and 3) the usefulness of the metrics as low quality filters. Through statistical analysis, we found that several metrics, especially Text Information Content, correlate well with human evaluation and that the average of all the metrics are roughly as effective as people to flag low quality instances. The implications of this finding are discussed. Finally, we propose possible applications of the metrics to improve tools for the administration of digital repositories.

Quality Ranking Algorithms for Knowledge Objects in Knowledge Management Systems

“…Some quality characteristics proposed in [28] include accuracy, provenance, completeness, consistency and coherence, timeliness and accessibility, and conformance to expectations. For instance, to evaluate the completeness of a knowledge attributes record (quality characteristic), we can check how many attributes have been filled with information (metric).…”

Reorienting Open Repositories to the Challenges of the Semantic Web: Experiences from FAO’s Contribution to the Resource Processing and Discovery Cycle in Repositories in the Agricultural Domain

“…Still, when different metadata standards and schemas are used across repositories this creates challenges in achieving interoperability [11] and Haslhofer and Klas [12] proposed metadata integration to solve this. However, Park and Lu [13] discovered that even in the use of a common metadata standard there was a divergence in what local metadata guidelines contained and what they represented.…”

Metadata for semantic and social applications<br />DC- 2008 Berlin

“…The critique interne is left to human evaluation, since it is impossible to develop automated tools to grasp evaluation criteria such as accuracy and conformance to expectations. Ochoa and Duval (2007) however propose to translate these and the other criteria from the Bruce and Hillmann framework into equations that can be automatically applied. Still, this approach only applies to metadata of textual resources and not to other types of unstructured data such as images.…”

“…And just as with manual sampling methods it only produces a "photograph" of the state of the metadata records at a specific moment in time. Ochoa and Duval (2007) point out a soft spot when they refer to metadata quality analysis as a "research activity with no practical implications in the functionality or performance of the digital repository. "…”