TDv2: A Novel Tree-Structured Decoder for Offline Mathematical Expression Recognition

Abstract

In recent years, tree decoders become more popular than LaTeX string decoders in the field of handwritten mathematical expression recognition (HMER) as they can capture the hierarchical tree structure of mathematical expressions. However previous tree decoders converted the tree structure labels into a fixed and ordered sequence, which could not make full use of the diversified expression of tree labels. In this study, we propose a novel tree decoder (TDv2) to fully utilize the tree structure labels. Compared with previous tree decoders, this new model does not require a fixed priority for different branches of a node during training and inference, which can effectively improve the model generalization capability. The input and output of the model make full use of the tree structure label, so that there is no need to find the parent node in the decoding process, which simplifies the decoding process and adds a prior information to help predict the node. We verified the effectiveness of each part of the model through comprehensive ablation experiments and attention visualization analysis. On the authoritative CROHME 14/16/19 datasets, our method achieves the state-of-the-art results.

A Step Toward an Automatic Handwritten Homework Grading System for Mathematics

“…To the best of our knowledge, we cannot find any other similar system that evaluates the entire mathematical derivation after recognition, so the direct performance comparison cannot be made. However, the ExpRate (expression recognition rates) before applying symbols replacement (depth = 0) in this study (69.18% for addition/subtraction and 64.39% for multiplication dataset) were on par with the 60-65% results in most of the recent literatures that propose mathematical expression recognition systems [27,28,29]. ExpRate after applying the symbols replacement from our system were increased to 89.31% and 87.90% for addition/subtraction and for multiplication dataset, respectively.…”

“…In the present, expression recognition rates (ExpRate) of encoderdecoder network are around 65% [27,28,29]. One of the limitations is the lack of large public dataset for the handwritten ME [1].…”

“…The state of the art for MER is based on encoder-decoder network, where the encoder transforms the entire image of ME into a sequence of vector representation, then the decoder converts the representation into a 1D latex syntax, or 2D expression tree structure. In the present, expression recognition rates (ExpRate) of encoder-decoder network are around 65% [27,28,29]. One of the limitations is the lack of large public dataset for the handwritten ME [1].…”

Language Model is Suitable for Correction of Handwritten Mathematical Expressions Recognition

“…utilized grammar constrained attention to transform the whole image into a parse tree. Wu et al (2022a) added thinking attention to tree decoder, assisted by pixel-level auxiliary loss to improve recognition of complex expressions. Wu et al (2022b) designed a structural string representation, attempting to utilize both language model and tree structure.…”

A survey on handwritten mathematical expression recognition: The rise of encoder-decoder and GNN models

“…Wu et al proposed TDv2 for offline HME recognition[92], which is free from the relation order of the input. They also added symbol-level and pixel-level auxiliary constraints while training their model.…”