Impact of Crowdsourcing OCR Improvements on Retrievability Bias

Abstract

Digitized document collections often suffer from OCR errors that may impact a document's readability and retrievability. We studied the effects of correcting OCR errors on the retrievability of documents in a historic newspaper corpus of a digital library. We computed retrievability scores for the uncorrected documents using queries from the library's search log, and found that the document OCR character error rate and retrievability score are strongly correlated. We computed retrievability scores for manually corrected versions of the same documents, and report on differences in their total sum, the overall retrievability bias, and the distribution of these changes over the documents, queries and query terms. For large collections, often only a fraction of the corpus is manually corrected. Using a mixed corpus, we assess how this mix affects the retrievability of the corrected and uncorrected documents. The correction of OCR errors increased the number of documents retrieved in all conditions. The increase contributed to a less biased retrieval, even when taking the potential lower ranking of uncorrected documents into account. CCS CONCEPTS• Information systems → Query log analysis; Evaluation of retrieval results;

Music in newspapers

“…Tokens with higher word-level confidence (which is positively influenced by successful matching to lists with known Dutch words) have higher retrievability scores [28]. Comparing OCR character error rates on newspapers from the 17th century vs. newspapers from the Second World War, the error rate is clearly higher for the 17th-century collection, negatively affecting retrievability of older articles [27].…”

“…Formal, large-scale OCR quality assessment is an emerging topic of interest at the KB. However, only small annotated ground truth datasets are available so far [27]. As discussed in [27][28][29], OCR quality impacts document retrievability.…”

“…However, only small annotated ground truth datasets are available so far [27]. As discussed in [27][28][29], OCR quality impacts document retrievability. Tokens with higher word-level confidence (which is positively influenced by successful matching to lists with known Dutch words) have higher retrievability scores [28].…”

“…Named entities are frequently present in the query logs of the Delpher portal [27]. Generally, knowledge of named entities is beneficial for linked and enriched data access.…”

“…As a means to scale up quality improvements at the OCR and NER linking levels, the KB currently investigates crowdsourcing possibilities [27,31]. This also will be a useful mechanism when seeking to engage music domain experts.…”