Performance Evaluation of a Mathematical Formula Recognition System with a large scale of printed formula images

Abstract

This paper shows the performance evaluation of the mathematical formula recognition system applied to a large scale of printed formula images, which has been developed in our laboratory. Our laboratory has collaborated with Michler of University of Essen, Germany, on project of "Retro-digitalization of mathematical journals, and their integration searchable digital libraries". In this project, two kinds of mathematical journals were scanned for digitalization. In this process, we cut out formulas manually from many pages, and created the answer database, "Ground-Truth." We report the results of performance evaluation with respect to the recognition of symbols, various types of sub-expressions and whole expressions of mathematical formulas.

Recognition and retrieval of mathematical expressions

“…The most common class of performance metrics for evaluation of math recognition systems are recognition rates, for complete expressions [29,110,163] and individual symbols [8,29,110,143]. Characterizations of layout structure accuracy have been measured using a variety of metrics; most simply, the number of symbols with the appropriate parent symbol, relationship, and depth in a symbol layout tree ('token placement'), and the number of baselines that contain the correct symbols [163].…”

“…(11,190 symbols), written by 10 different writers, and drawn from CRC Standard Mathematical Tables and Formulae [174]. Ashida et al [8] 1400 pages for symbol recognition data (43,495 typeset expressions), 700 pages for structure analysis (21,472 typeset expressions), taken from Archiv der Mathematik and Commentarii Mathematici Helvetici. Ground truth was created using automatic recognition followed by manual correction.…”

Issues in Performance Evaluation: A Case Study of Math Recognition

“…Ashida et al [6] Symbol recognition rate Chan and Yeung [10] Symbol recognition rate Expression recognition rate Operator recognition rate Integrated performance measure Garain and Chaudhuri [12] Global performance index Average performance index Kosmala et al [17] Computing time Okamoto et al [21] Expression recognition rate Character recognition rate Structure recognition rate Takiguchi et al [23] Character recognition rate Zanibbi et al [28] Baseline recognition rate Token placement rate Expression recognition rate…”

“…For lack of space, authors sometimes only report representative examples of confusions that are forgiven, e.g. [6].…”

“…Sometimes, blackbox evaluation is applied to individual system components; for example, symbol recognition may be evaluated separately from structure analysis [6][10] [12]. Executable code must be available for black box evaluation to be carried out.…”