Handwritten Mathematical Expression Recognition via Attention Aggregation based Bi-directional Mutual Learning

Bian, Xiaohang; Qin, Bo; Xin, Xiaozhe; Li, Jianwu; Su, Xiao; Wang, Yanfeng

doi:10.48550/arxiv.2112.03603

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With the advent of Transformer architecture [3], there has been a pivotal shift in the approach to solving OCR problems. Sequence-to-sequence models have gained popularity [7], where attention modules were used in conjunction with CNNs to extract features and BiLSTMs were used as the decoder [10]. For the transformer approach, the input is required to be a sequence, which is generally done by CNNs to transform a two-dimensional input into a one-dimensional sequence.…”

Section: Optical Character Recognitionmentioning

confidence: 99%

Doc2TeX: A Novel Approach To Convert Written Documents To LaTeX

Keshavaprasad,

Shishir,

Niranjan

et al. 2024

Preprint

View full text Add to dashboard Cite

Although Handwritten Mathematical Expression Recognition (HMER) tasks are a subset of the Optical Character Recognition (OCR) process, both of them have remained predominantly exclusive to each other. Owing to the widespread increase and progress in Artificial Intelligence and Machine Learning, both fields have benefited greatly, with state-of-the-art architectures being used. This domain is further accentuated by the fact that conversion of math-based handwritten literature to a digital format is extremely time-consuming and is often a manual procedure. HMER in particular, has remained a challenging problem ever since its realization, with several worldwide competitions such as the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) showcasing the fact that progress in this field is extremely slow. This is attributed to the fact that the Mathematical 'language' has an enormous amount of variables and spatial dependencies. To this end, the authors would like to present Doc2Tex, a novel approach in converting handwritten math-based documents to a digital format, with an expression rate (0-error) of up to 64.58%. The proposed approach aims to solve this issue by making use of two aspects: Implementing a model which leverages the Transformer architecture for both image understanding and wordpiece-level text generation (fine-tuned and vanilla), and using a suitable probability indicator to differentiate between mathematical expressions and the English language. The approach put forth in this paper encompasses a wide variety of mathematical symbols and expressions in its vocabulary including complex symbols such as those from calculus and statistical probability domains. The inclusion of a wide variety of complex symbols in the vocabulary has not been done in most of the previous literary works examined by the authors. Metrics of the approach presented in this paper compared to the meta in HMER and OCR tasks have been provided.

show abstract

Section: Optical Character Recognitionmentioning

confidence: 99%