Syntax-Aware Network for Handwritten Mathematical Expression Recognition

Yuan, Ye; Liu, Xiao; Dikubab, Wondimu; Liu, Hui; Ji, Zhilong; Wu, Zhongqin; Bai, Xiang

doi:10.1109/cvpr52688.2022.00451

Cited by 44 publications

(19 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To the best of our knowledge, we cannot find any other similar system that evaluates the entire mathematical derivation after recognition, so the direct performance comparison cannot be made. However, the ExpRate (expression recognition rates) before applying symbols replacement (depth = 0) in this study (69.18% for addition/subtraction and 64.39% for multiplication dataset) were on par with the 60-65% results in most of the recent literatures that propose mathematical expression recognition systems [27,28,29]. ExpRate after applying the symbols replacement from our system were increased to 89.31% and 87.90% for addition/subtraction and for multiplication dataset, respectively.…”

Section: Expratementioning

confidence: 52%

“…In the present, expression recognition rates (ExpRate) of encoderdecoder network are around 65% [27,28,29]. One of the limitations is the lack of large public dataset for the handwritten ME [1].…”

Section: Structure Of Cnnmentioning

confidence: 99%

“…The state of the art for MER is based on encoder-decoder network, where the encoder transforms the entire image of ME into a sequence of vector representation, then the decoder converts the representation into a 1D latex syntax, or 2D expression tree structure. In the present, expression recognition rates (ExpRate) of encoder-decoder network are around 65% [27,28,29]. One of the limitations is the lack of large public dataset for the handwritten ME [1].…”

Section: Tablementioning

confidence: 99%

See 2 more Smart Citations

A Step Toward an Automatic Handwritten Homework Grading System for Mathematics

Chaowicharat

Dejdumrong

2023

ITC

View full text Add to dashboard Cite

An automatic system that helps teachers and students verify the correctness of handwritten derivation in mathematics homework is proposed. The system acquires input image containing handwritten mathematical derivation. In our preliminary study, the system that comprises only mathematical expression recognition (MER) and computer algebra system (CAS) did not perform well due to high misrecognition rate. Therefore, our study focuses on fixing the misrecognized symbols by using symbols replacement and the surrounding information. If all the original mathematical expressions (MEs) in the derivation sequence are already equivalent, the derivation is marked as “correct”. Otherwise, the symbols with low recognition confidence will be replaced by other possible candidates to maximize the number of equivalent MEs in that derivation. If there is none of symbols replacement that makes every line equivalent, the derivation is marked as “incorrect”. The recursive expression tree comparison was applied to report the types of mistake for those problems marked as incorrect. Finally, the performance of the system was evaluated by the digitally generated dataset of 6,000 handwritten mathematical derivations. The results showed that the symbols replacement improve the F1-score of derivation step marking from 69.41 to 95.95 % for the addition/ subtraction dataset and from 61.45 to 89.95 % for the multiplication dataset when compared to the case of using raw recognized string without symbols replacement.

show abstract

Section: Expratementioning

confidence: 52%

Section: Structure Of Cnnmentioning

confidence: 99%

Section: Tablementioning

confidence: 99%

See 1 more Smart Citation

A Step Toward an Automatic Handwritten Homework Grading System for Mathematics

Chaowicharat

Dejdumrong

2023

ITC

View full text Add to dashboard Cite

show abstract

“…At each step, a parent and child node pair was generated and the relation between parent node and child node reflects the structure type. Yuan et al [39] firstly incorporate syntax information into the encoderdecoder, which achieved higer recognition accuracy while taking into account speed. Li et al [17] design a weakly-supervised counting module and jointly optimizes HMER task and symbol counting task.…”

Section: Related Workmentioning

confidence: 99%

“…As shown in tab. 1 and 2, we compare our prosposed method with DWAP [42], DWAP-TD [46], BTTR [51], ABM [3], SAN [39] and CAN [17] on HME100K dataset. It is clear to notice that SAM-DWAP and SAM-CAN achieves the best performance.…”

Section: Comparison With State-of-the-artmentioning

confidence: 99%

Semantic Graph Representation Learning for Handwritten Mathematical Expression Recognition

Liu,

Yuan,

et al. 2023

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Handwritten mathematical expression recognition (HMER) has attracted extensive attention recently. However, current methods cannot explicitly study the interactions between different symbols, which may fail when faced similar symbols. To alleviate this issue, we propose a simple but efficient method to enhance semantic interaction learning (SIL). Specifically, we firstly construct a semantic graph based on the statistical symbol co-occurrence probabilities. Then we design a semantic aware module (SAM), which projects the visual and classification feature into semantic space. The cosine distance between different projected vectors indicates the correlation between symbols. And jointly optimizing HMER and SIL can explicitly enhances the model's understanding of symbol relationships. In addition, SAM can be easily plugged into existing attention-based models for HMER and consistently bring improvement. Extensive experiments on public benchmark datasets demonstrate that our proposed module can effectively enhance the recognition performance. Our method achieves better recognition performance than prior arts on both CROHME and HME100K datasets.

show abstract