From Documents to Data: A Framework for Total Corpus Quality

Abstract

As large corpora of digitized text and novel methodologies become increasingly available, researchers are rediscovering textual data’s potential fruitfulness for inquiries into social and cultural phenomena. While textual corpora show great promise to enrich our knowledge of the social, avoiding problems related to data quality remains a challenge to related empirical research. Hence, evaluating the quality of a corpus will be pivotal for future social science inquiries. We propose a conceptual framework for total corpus quality incorporating three important dimensions—total corpus error, corpus comparability, and corpus reproducibility—impacting the validity and reliability of inferences drawn from textual data. This framework provides insights toward evaluating and improving studies based on large-scale textual analyses. We employ a case study to exemplify and discuss how researchers can identify and measure the three proposed quality dimensions for any given corpus.

Meta-dominance analysis – A tool for the assessment of the quality of digital behavioural data

“…The relative error is therefore only able to identify groups of seemingly errorprone factors in the research process, rather than definite sources of error. However, although an absolute measure of error (Meyer & Mittag, 2021;Hurtado Bodell et al, 2022) would be better suited to assess DBD quality, such an approach must first identify a true effect from which different error dimensions deviate. This is impossible in most practical cases.…”