Quantifying the impact of dirty OCR on historical text analysis: Eighteenth Century Collections Online as a case study

Abstract

This article aims to quantify the impact optical character recognition (OCR) has on the quantitative analysis of historical documents. Using Eighteenth Century Collections Online as a case study, we first explore and explain the differences between the OCR corpus and its keyed-in counterpart, created by the Text Creation Partnership. We then conduct a series of specific analyses common to the digital humanities: topic modelling, authorship attribution, collocation analysis, and vector space modelling. The article concludes by offering some preliminary thoughts on how these conclusions can be applied to other datasets, by reflecting on the potential for predicting the quality of OCR where no ground-truth exists.

Assessing the Impact of OCR Quality on Downstream NLP Tasks

“…This is an instance of intrinsic OCR evaluation, where we only rely on the OCR model to assess it(self). Such assessments are unsatisfactory because they might not be comparable when the software/provider changes and provide no indication on how the OCR quality influences other tasks or is related to other, external data or systems (Hill and Hengchen, 2019). This is the broader scope of extrinsic OCR evaluations.…”

“…Studies have considered information access and retrieval (Traub et al, 2018), authorship attribution (Franzini et al, 2018), named entity recognition (Hamdi et al, 2019), and topic modelling (Nelson, 2020;Mutuvi et al, 2018). Recently (Hill and Hengchen, 2019) compared different tasks on a corpus in English: topic modelling, collocation analysis, authorship attribution and vector space modelling. From this study, a critical OCR quality threshold between 70 and 80% emerged, where most tasks perform very poorly below this threshold, good results are achieved above it, and varying results are achieved within, according to the task at hand.…”

“…Another element to be explored is the impact of time, and consequently of the combined effects of linguistic change and OCR quality on the application of tools usually trained on contemporary languages. Lastly, the range of tasks which are considered in previous work is limited, with comparisons across tasks attempted in a single, seminal paper (Hill and Hengchen, 2019). In this work, we start addressing these research questions by considering a larger set of tasks and utilizing text drawn from a source which poses many challenges for OCR: historic newspapers (Pletschacher et al, 2014).…”

“…We find that, while quality bands 1 and, to a lesser degree 2, still maintain a good fidelity with their human-corrected counterparts, this is not the case for bands 3 and, particularly, 4. The issue is not as much that OCR topic models became meaningless but, more subtly, that they retain their interpretability (Hill and Hengchen, 2019) while becoming substantially different from what they would be using clean texts. − ∑ t∈V i p i (t)log[p i (t)].…”

Endangered Languages are not Low-Resourced!

“…If some NLP resource does not exist for a language, stop complaining about how low-resourced it is, get up and gather the data. Of course, there are always exceptions when gathering the data required for a large language might not be a walk in a park such as when dealing with historical data [11]. And it is true that even resources for non-endangered languages can be noisy [19].…”