A framework for the assessment of text extraction algorithms on complex colour images

Abstract

The availability of open, ground-truthed datasets and clear performance metrics is a crucial factor in the development of an application domain. The domain of colour text image analysis (real scenes, Web and spam images, scanned colour documents) has traditionally suffered from a lack of a comprehensive performance evaluation framework. Such a framework is extremely difficult to specify, and corresponding pixel-level accurate information tedious to define. In this paper we discuss the challenges and technical issues associated with developing such a framework. Then, we describe a complete framework for the evaluation of text extraction methods at multiple levels, provide a detailed ground-truth specification and present a case study on how this framework can be used in a real-life situation.

Document Analysis Research in the Year 2021

“…Past experience has shown that attempts to "coerce" a community into using a single set of conventions does not work; at best, it contributes to locking it into a limited subset of possible uses, stifling creativity. This is contradictory to the standpoint we have taken with respect to ground-truth (or rather lack thereof [6,10,14,2]) and our preferring the term interpretation. This clearly advocates for as open as possible ways of representing data, keeping in mind, however, that abandoning any kind of imposed structure may make it impossible to realize our vision.…”

ICDAR 2013 Robust Reading Competition

“…The ground-truth specification is presented in [12] and captures information at different levels, from character parts up to text lines, and from pixel level labelling up to bounding boxes and transcriptions. The default ground truth format is XML as described in [12]. For simplicity we extracted individual aspects of the ground truth for the different tasks.…”

“…One is the standard pixel level precision and recall metric, which we include only for completeness and backward compatibility but we do not use for the final ranking as it has known problems. The primary evaluation scheme we used is the one described in [12], as it is designed taking into account the final objective which is segmentation for recognition. The question of segmentation is not only how many pixels are misclassified but which ones.…”

“…In the case of text segmentation, we are evaluating results in two ways: our primary measure is based on the methodology detailed in [12], where quantitative evaluation is tightly connected with the quality of the result in terms of the integrity of the individual atoms on the page. Given a set of labeled connected components, this method reports the percentages of ground truth atoms that are "Well Segmented", "Merged" and "Broken" in the segmentation results.…”

Text Extraction from Scene Images Through Local Binary Pattern and Business Features Based Color Image Segmentation

“…On the other hand, for the test set, we apply the evaluation criteria of Clavelli et al [10]. They proposed a number of measures to assess the segmentation quality of each of the text-parts defined in the ground truth.…”