Content Profiling for Preservation: Improving Scale, Depth and Quality

Kulmukhametov, Artur; Becker, Christoph

doi:10.1007/978-3-319-12823-8_1

Cited by 4 publications

(8 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We adopted a very defensive approach to rule definition to avoid distorting errors, so numerous conflicting patterns remain for which no resolution is specified. Still, similar to the findings in Kulmukhametov & Becker (2014), a small number of rules is highly effective in improving data quality. The effect on the ratio of conflicted values shows that residual conflicts are small.…”

Section: Table 4 Exemplary Conflicts In the Data Set And Their Resolusupporting

confidence: 64%

“…This is essential for subsequent analysis since some segments contain large shares of conflicted items. The implemented mechanisms go beyond the data cleansing performed by Jackson (2012) and build on earlier work (Kulmukhametov & Becker, 2014). An analysis of conflict patterns supported the declarative formulation of a relatively small rule set in the market spreadsheet so that conflict resolution is extensible and fully integrated.…”

Section: Input Preparation and Configurationmentioning

confidence: 99%

“…Identification tools are not perfect (Holden, 2012) and will often deliver conflicting results (Kulmukhametov & Becker, 2014;Kumaravel, Dearborn, Witt, & Kuang, 2016). However, these conflicts occur in patterns according to a power law distribution, so we can identify frequently occurring conflicts that can be explained and resolve them using rule-based techniques (Kulmukhametov & Becker, 2014 To calculate technology lifecycles, we also need a specification of markets that partitions the set into meaningful segments, such as the market of known image formats, and provides product release years to enable a unified mapping of multiple format statistics onto an age-adjusted curve. We compiled a spreadsheet that combines multiple formats into groups to define market segments of interest.…”

Section: Figure 4 Uk Web Archive Volume Per Yearmentioning

confidence: 99%

See 2 more Smart Citations

Format technology lifecycle analysis

Duretec

Becker

2017

Asso for Info Science & Tech

Self Cite

View full text Add to dashboard Cite

The lifecycles of format technology have been a defining concern for digital stewardship research and practice. However, little evidence exists to provide robust methods for assessing the state of any given format technology and describing its evolution over time.This article introduces relevant models from diffusion theory and market research and presents a replicable analysis method to compute models of technology evolution. Data cleansing and the combination of multiple data sources enable the application of non-linear regression to estimate the parameters of the Bass diffusion model on format technology market lifecycles.Through its application to a longitudinal dataset from the UK web archive, we demonstrate that the method produces reliable results and show that the Bass model can be used to describe format lifecycles. By analyzing adoption patterns across market segments, new insights are inferred about how the diffusion of formats and products such as applications occurs over time. The analysis provides a stepping stone to a more robust and evidence-based approach to model technology evolution.

show abstract

Section: Table 4 Exemplary Conflicts In the Data Set And Their Resolusupporting

confidence: 64%

Section: Input Preparation and Configurationmentioning

confidence: 99%

Section: Figure 4 Uk Web Archive Volume Per Yearmentioning

confidence: 99%

See 1 more Smart Citation

Format technology lifecycle analysis

Duretec

Becker

2017

Asso for Info Science & Tech

Self Cite

View full text Add to dashboard Cite

show abstract

“…To diversify aspects such as the number of text snippets and their order and implementation possibilities, a set of model transformations is de ned that operate on the PIM and PSM level. To achieve a realistic dataset in terms of the feature distributions, we use the content pro ling tool C3PO [14] to sample initial real world distributions from the Govdocs data set. Where feature distribution data is not available, the transformation falls back to its default distribution.…”

Section: Instantiating the Data Generation Framework For This Benchmarkmentioning

confidence: 99%

A Text Extraction Software Benchmark Based on a Synthesized Dataset

Duretec¹,

Rauber²,

Becker³

2017

2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL)

Self Cite

View full text Add to dashboard Cite

Text extraction plays an important function for data processing work ows in digital libraries. For example, it is a crucial prerequisite for evaluating the quality of migrated textual documents. Complex le formats make the extraction process error-prone and have made it very challenging to verify the correctness of extraction components. Based on digital preservation and information retrieval scenarios, three quality requirements in terms of e ectiveness of text extraction tools are identi ed: 1) is a certain text snippet correctly extracted from a document, 2) does the extracted text appear in the right order relative to other elements and, 3) is the structure of the text preserved. A number of text extraction tools is available ful lling these three quality requirements to various degrees. However, systematic benchmarks to evaluate those tools are still missing, mainly due to the lack of datasets with accompanying ground truth. e contribution of this paper is twofold. First we describe a dataset generation method based on model driven engineering principles and use it to synthesize a dataset and its ground truth directly from a model. Second, we de ne a benchmark for text extraction tools and complete an experiment to calculate performance measures for several tools that cover the three quality requirements. e results demonstrate the bene ts of the approach in terms of scalability and e ectiveness in generating ground truth for content and structure of text elements.

show abstract

“…However, the accuracy and correctness of the tools vary [19,20], and they frequently disagree on such questions as which format is this object encoded in?. As a consequence, they produce contradicting outputs [23,24]. Another source of contradiction is evolving metadata [25] when information and metadata standards change over time.…”

Section: Introductionmentioning

confidence: 99%

Improving data quality in large-scale repositories through conflict resolution

2021

Self Cite

View full text Add to dashboard Cite

Digital repositories rely on technical metadata to manage their objects. The output of characterization tools is aggregated and analyzed through content profiling. The accuracy and correctness of characterization tools vary; they frequently produce contradicting outputs, resulting in metadata conflicts. The resulting metadata conflicts limit scalable preservation risk assessment and repository management. This article presents and evaluates a rule-based approach to improving data quality in this scenario through expert-conducted conflict resolution. We characterize the data quality challenges and present a method for developing conflict resolution rules to improve data quality. We evaluate the method and the resulting data quality improvements in an experiment on a publicly available document collection. The results demonstrate that our approach enables the effective resolution of conflicts by producing rules that reduce the number of conflicts in the data set from 17 to 3%. This replicable method for presents a significant improvement in content profiling technology for digital repositories, since the enhanced data quality can improve risk assessment and preservation management in digital repository systems.

show abstract

Content Profiling for Preservation: Improving Scale, Depth and Quality

Cited by 4 publications

References 7 publications

Format technology lifecycle analysis

Format technology lifecycle analysis

A Text Extraction Software Benchmark Based on a Synthesized Dataset

Improving data quality in large-scale repositories through conflict resolution

Contact Info

Product

Resources

About