Multi-modal Attention Network for Handwritten Mathematical Expression Recognition

Abstract

In this paper, we propose a novel stroke constrained attention network (SCAN) which treats stroke as the basic unit for encoder-decoder based online handwritten mathematical expression recognition (HMER). Unlike previous methods which use trace points or image pixels as basic units, SCAN makes full use of stroke-level information for better alignment and representation. The proposed SCAN can be adopted in both single-modal (online or offline) and multi-modal HMER. For single-modal HMER, SCAN first employs a CNN-GRU encoder to extract point-level features from input traces in online mode and employs a CNN encoder to extract pixel-level features from input images in offline mode, then use stroke constrained information to convert them into online and offline stroke-level features. Using stroke-level features can explicitly group points or pixels belonging to the same stroke, therefore reduces the difficulty of symbol segmentation and recognition via the decoder with attention mechanism. For multi-modal HMER, other than fusing multi-modal information in decoder, SCAN can also fuse multi-modal information in encoder by utilizing the stroke based alignments between online and offline modalities. The encoder fusion is a better way for combining multi-modal information as it implements the information interaction one step before the decoder fusion so that the advantages of multiple modalities can be exploited earlier and more adequately when training the encoder-decoder model. Evaluated on a benchmark published by CROHME competition, the proposed SCAN achieves the state-of-the-art performance.

Improving Attention-Based Handwritten Mathematical Expression Recognition with Scale Augmentation and Drop Attention

“…Systems I to VII were participating systems except for system III because of using private training data. We do not present results from [21], [22] and [23] because their models were trained on handwriting trajectory data. A trajectory provides handwriting order information, which is useful for distinguishing visually similar symbols (e.g., "α" and "a").…”

TDv2: A Novel Tree-Structured Decoder for Offline Mathematical Expression Recognition

“…Hong (Hong et al 2019) employed residual connection in BiRNN to improve feature extraction. Besides, multi-modal learning was also introduced into HMER with both online and offline information for better encoding (Wang et al 2019) and decoding (Wang et al 2021). There are also approaches which apply data augmentation (Le and Nakagawa 2017;Le, Indurkhya, and Nakagawa 2019;Li et al 2020).…”

Syntax-Aware Network for Handwritten Mathematical Expression Recognition

“…Most HMER methods extensively adopt the sequence-to-sequence approach. The authors in [8,15,16,23,27,29,39,40,40,43,46] proposed an attention-based sequence-to-sequence model to convert the handwritten mathematical expression images into rep- resentational markup language LaTeX. Recently, Wu et al [31] designed a graph-to-graph(G2G) model that explores the HMEs structural relationship of the input formula and output markup, which significantly improve the performance.…”