The Challenge of Test Data Quality in Data Processing

Becker, Christoph; Duretec, Kresimir; Rauber, Andreas

doi:10.1145/3012004

Cited by 8 publications

(7 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…e development of high quality test datasets is a key concern for any kind of data processing evaluation [9] [5]. is has been a well recognized challenge for years in domains such as digital preservation [22].…”

Section: Discussionmentioning

confidence: 99%

“…Other datasets with ground truth include CleanEval 4 and L3S-GN1 5 . Unfortunately, these datasets cover only web page documents and are tailored for narrow needs of distinguishing real web text content from boilerplate text.…”

Section: Background 21 Text Extraction Evaluation and Test Datasetsmentioning

confidence: 99%

“…In some cases, datasets annotations are done by other tools such as the one developed and used by Pasternack et al [24]. However, since such annotations are computed by tools that are not evaluated themselves, their suitability for evaluation purposes is questionable [4,5]. is has been recognized by Pasternack el al.…”

Section: Background 21 Text Extraction Evaluation and Test Datasetsmentioning

confidence: 99%

“…However, these criteria do not cover speci c technical aspects such as coverage and representativeness. is lack of test data quality models is a recognized problem for this domain [5], and more research is needed to de ne a model of quality characteristics and metrics.…”

Section: So Ware Benchmarking and So Ware Testingmentioning

confidence: 99%

See 3 more Smart Citations

A Text Extraction Software Benchmark Based on a Synthesized Dataset

Duretec¹,

Rauber²,

Becker³

2017

2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL)

Self Cite

View full text Add to dashboard Cite

Text extraction plays an important function for data processing work ows in digital libraries. For example, it is a crucial prerequisite for evaluating the quality of migrated textual documents. Complex le formats make the extraction process error-prone and have made it very challenging to verify the correctness of extraction components. Based on digital preservation and information retrieval scenarios, three quality requirements in terms of e ectiveness of text extraction tools are identi ed: 1) is a certain text snippet correctly extracted from a document, 2) does the extracted text appear in the right order relative to other elements and, 3) is the structure of the text preserved. A number of text extraction tools is available ful lling these three quality requirements to various degrees. However, systematic benchmarks to evaluate those tools are still missing, mainly due to the lack of datasets with accompanying ground truth. e contribution of this paper is twofold. First we describe a dataset generation method based on model driven engineering principles and use it to synthesize a dataset and its ground truth directly from a model. Second, we de ne a benchmark for text extraction tools and complete an experiment to calculate performance measures for several tools that cover the three quality requirements. e results demonstrate the bene ts of the approach in terms of scalability and e ectiveness in generating ground truth for content and structure of text elements.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Background 21 Text Extraction Evaluation and Test Datasetsmentioning

confidence: 99%

Section: Background 21 Text Extraction Evaluation and Test Datasetsmentioning

confidence: 99%

Section: So Ware Benchmarking and So Ware Testingmentioning

confidence: 99%

See 2 more Smart Citations

A Text Extraction Software Benchmark Based on a Synthesized Dataset

Duretec¹,

Rauber²,

Becker³

2017

2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In order to establish whether a piece of DF software functions correctly, it must be validated, where Guo et al's., (2009, p.13) definition states 'validation is the confirmation by examination and the provision of objective evidence that a tool, technique or procedure functions correctly and as intended'. In essence, a level of 'functional correctness' needs to be determined and assessed before a tool should be utilised in a live investigation (SWGDE, 2014;SWGDE, 2017;Becker et al, 2017), yet in practice it can be difficult to do this. The Forensic Science Regulator (2015) provides direction in the form of guidelines designed to support the the development of methods for validating forensic software with acknowledgement of the need to embed validation into laboratory practices in order to adhere to the ISO 17025 standard.…”

Section: Digital Forensic Softwarementioning

confidence: 99%

“I couldn't find it your honour, it mustn't be there!” – Tool errors, tool limitations and user error in digital forensics

Horsman

2018

Science & Justice

View full text Add to dashboard Cite

The field of digital forensics maintains significant reliance on the software it uses to acquire and investigate forms of digital evidence. Without these tools, analysis of digital devices would often not be possible. Despite such levels of reliance, techniques for validating digital forensic software are sparse and research is limited in both volume and depth. As practitioners pursue the goal of producing robust evidence, they face the onerous task of both ensuring the accuracy of their tools and, their effective use. Whilst tool errors provide one issue, establishing a tool's limitations also provides an investigatory challenge leading the potential for practitioner user-error and ultimately a grey area of accountability. This article debates the problems surrounding digital forensic tool usage, evidential reliability and validation.

show abstract

Proposal for an Evaluation Framework for Compliance Checkers for Long-Term Digital Preservation

Ferro

2017

Communications in Computer and Information Science

View full text Add to dashboard Cite

The Challenge of Test Data Quality in Data Processing

Cited by 8 publications

References 13 publications

A Text Extraction Software Benchmark Based on a Synthesized Dataset

A Text Extraction Software Benchmark Based on a Synthesized Dataset

“I couldn't find it your honour, it mustn't be there!” – Tool errors, tool limitations and user error in digital forensics

Proposal for an Evaluation Framework for Compliance Checkers for Long-Term Digital Preservation

Contact Info

Product

Resources

About