Object count/area graphs for the evaluation of object detection and segmentation algorithms

Abstract

Evaluation of object detection algorithms is a non-trivial task: a detection result is usually evaluated by comparing the bounding box of the detected object with the bounding box of the ground truth object. The commonly used precision and recall measures are computed from the overlap area of these two rectangles. However, these measures have several drawbacks: they don't give intuitive information about the proportion of the correctly detected objects and the number of false alarms, and they cannot be accumulated across multiple images without creating ambiguity in their interpretation. Furthermore, quantitative and qualitative evaluation is often mixed resulting in ambiguous measures.In this paper we propose a new approach which tackles these problems. The performance of a detection algorithm is illustrated intuitively by performance graphs which present object level precision and recall depending on constraints on detection quality. In order to compare different detection algorithms, a representative single performance value is computed from the graphs. The influence of the test database on the detection performance is illustrated by performance/generality graphs. The evaluation method can be applied to different types of object detection algorithms. It has been tested on different text detection algorithms, among which are the participants of the ICDAR 2003 text detection competition.

ICDAR 2011 Robust Reading Competition Challenge 2: Reading Text in Scene Images

“…The main question when choosing a method for scene text detection particularly is how to deal with under and over segmentation errors. In this competition, we employ the method by Wolf et al [15] that is specifically designed to evaluate scene text detection approaches. We used the DetEval 2 evaluation software with default parameters for evaluating the competing methods.…”

“…The concept of scene text recognition challenge is similar to ICDAR 2003 and ICDAR 2005 Robust Reading competitions. Some problems were reported about the dataset (slightly larger bounding boxes, inconsistent definition of a "word" for instance whether a hyphen breaks the words or not) as well as the evaluation scheme (handling of one-to-many and manyto-one matches) used in the previous competitions [15]. We created ground-truth of the ICDAR 2003 Robust Reading competition dataset from scratch and adapted a new evaluation scheme [15] to resolve these issues.…”

Automatic Text Localisation in Scanned Comic Books

“…We used (Wolf and Jolion, 2006) to evaluate our results following the approach of the ICDAR competition 2011 (Karatzas et al, 2011). We used the area precision thresholds proposed in the original publication: t p = 0.4 and we decreased the area recall threshold from t r = 0.8 to t r = 0.6 in order to be more flexible with accent and punctuation miss detections (e.g.À,È,É, ..., !, ?)…”

On Combining Multiple Segmentations in Scene Text Recognition

“…Using the ICDAR 2011 competition evaluation scheme [17], the method achieves recall 67.6%, precision 81.1% and fmeasure 75.4% in text localization (see Figure 6 for sample outputs). This represents a significant 4 percentage point improvement over the best published result (the ICDAR 2011 Robust Reading competition winner [14]) and a 7 percentage point improvement over the NM12 method [11], which demonstrates the impact of novel contributions presented in this paper.…”