Stroke-Based Performance Metrics for Handwritten Mathematical Expressions

Abstract

Abstract-Evaluating mathematical expression recognition involves a complex interaction of input primitives (e.g. pen/finger strokes), recognized symbols, and recognized spatial structure. Existing performance metrics simplify this problem by separating the assessment of spatial structure from the assessment of symbol segmentation and classification. These metrics do not characterize the overall accuracy of a penbased mathematics recognition, making it difficult to compare math recognition algorithms, and preventing the use of machine learning algorithms requiring a criterion function characterizing overall system performance. To address this problem, we introduce performance metrics that bridge the gap from handwritten strokes to spatial structure. Our metrics are computed using bipartite graphs that represent classification, segmentation and spatial structure at the stroke level. Overall correctness of an expression is measured by counting the number of relabelings of nodes and edges needed to make the bipartite graph for a recognition result match the bipartite graph for ground truth. This metric may also be used with other primitive types (e.g. image pixels).

Recognition of on-line handwritten mathematical expressions using 2D stochastic context-free grammars and hidden Markov models

“…The main problem of the EMERS metric is that the representation ambiguity of coded mathematical expressions can produce different trees of the same expression. In [41], the authors proposed a representation that seems to cope with this ambiguity and they also presented their corresponding performance metrics. However, the representation required precise information of the structure of the expression at stroke level, and the MathBrush corpus annotation had not completely and explicitly this information.…”

“…Evaluation of mathematical expression recognition systems is a difficult problem [5,19], and several metrics have been proposed [30,41,1]. The EMERS metric [30] is a dissimilitude value computed as the tree edit distance between the tree representation of the obtained expression and the reference expression, ant it is not normalized.…”

“…Evaluation and comparison of the structural analysis in mathematical expression recognition has hitherto been difficult because: first, most of the proposals used private data, and second, there has been a lack of standard performance evaluation measures [5,19]. Lately, some public large corpora have been developed [23,32] and also new metrics have been proposed [1,30,41]. Recently a Competition on Recognition of Online Handwritten Mathematical Expressions 1 (CROHME 2011) has been proposed [25].…”

An Image-Based Measure for Evaluation of Mathematical Expression Recognition

“…EMERS computes an edit distance using the tree representation of the ME. Zanibbi et al [11] defined a set of performance metrics at different levels based on bipartite graph representation: that different metrics seem to provide a canonical representation, but it is not detailed in the paper and no experimentation is reported.…”

“…Given that the recognition of mathematical symbols can be stated as a regular classification problem, the classification error rate of individual symbols is usually provided as a performance measure. However the recognition of the structural relation between mathematical symbols, which can be seen a parsing problem, requires more sophisticated evaluation methods [8,11].…”