Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition

2018 24th International Conference on Pattern Recognition (ICPR)

Dai

2018

Self Cite

134

Handwritten mathematical expression recognition is a challenging problem due to the complicated two-dimensional structures, ambiguous handwriting input and variant scales of handwritten math symbols. To settle this problem, recently we propose the attention based encoder-decoder model that recognizes mathematical expression images from two-dimensional layouts to one-dimensional LaTeX strings. In this study, we improve the encoder by employing densely connected convolutional networks as they can strengthen feature extraction and facilitate gradient propagation especially on a small training set. We also present a novel multi-scale attention model which is employed to deal with the recognition of math symbols in different scales and restore the fine-grained details dropped by pooling operations. Validated on the CROHME competition task, the proposed method significantly outperforms the state-of-theart methods with an expression recognition accuracy of 52.8% on CROHME 2014 and 50.1% on CROHME 2016, by only using the official training dataset.

Section: B Decodermentioning

confidence: 99%

Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition

2018 24th International Conference on Pattern Recognition (ICPR)

Dai

2018

Self Cite

134

“…Besides, [30] adopted residual connection in encoder and a transition probability matrix in decoder. As for the offline approach, [11] utilized a WAP model, which adopted CNNbased encoder to extract features from static images. [12] proposed a coarse-tofine attention to improve efficiency.…”

Section: Attention Based Encoder-decoder Approaches For Hmermentioning

confidence: 99%

“…For online HMER, [9,10] treat the handwritten mathematical expression (HME) as a point sequence and extract point-level features from input traces. While for offline HMER, [11,12] take the HME as a static image and extract pixel-level features from the input image. Benefiting from rich dynamic (spatial and temporal) information which is extremely helpful for handwritten recognition, online HMER tends to meet fewer difficulties caused by ambiguous handwriting.…”

Section: Introductionmentioning

confidence: 99%

Multi-modal Attention Network for Handwritten Mathematical Expression Recognition

Wang

2019 International Conference on Document Analysis and Recognition (ICDAR)

et al. 2019

Self Cite

In this paper, we propose a novel stroke constrained attention network (SCAN) which treats stroke as the basic unit for encoder-decoder based online handwritten mathematical expression recognition (HMER). Unlike previous methods which use trace points or image pixels as basic units, SCAN makes full use of stroke-level information for better alignment and representation. The proposed SCAN can be adopted in both single-modal (online or offline) and multi-modal HMER. For single-modal HMER, SCAN first employs a CNN-GRU encoder to extract point-level features from input traces in online mode and employs a CNN encoder to extract pixel-level features from input images in offline mode, then use stroke constrained information to convert them into online and offline stroke-level features. Using stroke-level features can explicitly group points or pixels belonging to the same stroke, therefore reduces the difficulty of symbol segmentation and recognition via the decoder with attention mechanism. For multi-modal HMER, other than fusing multi-modal information in decoder, SCAN can also fuse multi-modal information in encoder by utilizing the stroke based alignments between online and offline modalities. The encoder fusion is a better way for combining multi-modal information as it implements the information interaction one step before the decoder fusion so that the advantages of multiple modalities can be exploited earlier and more adequately when training the encoder-decoder model. Evaluated on a benchmark published by CROHME competition, the proposed SCAN achieves the state-of-the-art performance.

“…As for the unseen 16,079 Chinese characters, we choose 2,000 characters as the validation set and 14,079 characters as the testing set. We also employ an ensemble method during testing procedure [13] because the performances vary severely due to the small training set. We illustrate the performance in Fig.…”

Section: Experiments On Recognition Of Unseen Charactersmentioning

confidence: 99%

Radical Analysis Network for Zero-Shot Learning in Printed Chinese Character Recognition

2018 IEEE International Conference on Multimedia and Expo (ICME)

Zhu

et al. 2018

Self Cite

Chinese characters have a huge set of character categories, more than 20,000 and the number is still increasing as more and more novel characters continue being created. However, the enormous characters can be decomposed into a compact set of about 500 fundamental and structural radicals. This paper introduces a novel radical analysis network (RAN) to recognize printed Chinese characters by identifying radicals and analyzing two-dimensional spatial structures among them. The proposed RAN first extracts visual features from input by employing convolutional neural networks as an encoder. Then a decoder based on recurrent neural networks is employed, aiming at generating captions of Chinese characters by detecting radicals and two-dimensional structures through a spatial attention mechanism. The manner of treating a Chinese character as a composition of radicals rather than a single character class largely reduces the size of vocabulary and enables RAN to possess the ability of recognizing unseen Chinese character classes, namely zero-shot learning.