A general framework for the recognition of online handwritten graphics

Abstract

We propose a new framework for the recognition of online handwritten graphics. Three main features of the framework are its ability to treat symbol and structural level information in an integrated way, its flexibility with respect to different families of graphics, and means to control the tradeoff between recognition effectiveness and computational cost. We model a graphic as a labeled graph generated from a graph grammar. Non-terminal vertices represent subcomponents, terminal vertices represent symbols, and edges represent relations between subcomponents or symbols. We then model the recognition problem as a graph parsing problem: given an input stroke set, we search for a parse tree that represents the best interpretation of the input. Our graph parsing algorithm generates multiple interpretations (consistent with the grammar) and then we extract an optimal interpretation according to a cost function that takes into consideration the likelihood scores of symbols and structures. The parsing algorithm consists in recursively partitioning the stroke set according to structures defined in the grammar and it does not impose constraints present in some previous works (e.g. stroke ordering). By avoiding such constraints and thanks to the powerful representativeness of graphs, our approach can be adapted to the recognition of different graphic notations. We show applications to the recognition of mathematical expressions and flowcharts. Experimentation shows that our method obtains state-of-the-art accuracy in both applications.

Graph-to-Graph: Towards Accurate and Interpretable Online Handwritten Mathematical Expression Recognition

“…Compared with string grammar, graph grammar can better represent the hierarchical structure of formula. Several graph grammar based methods have been investigated for recognizing formula as graph (Lavirotte 1997;Julca-Aguilar et al 2020). There are also some approaches (Mahdavi et al and target graph is construct on the SLT.…”

“…In addition to image-to-markup methods, grammatical methods (Julca-Aguilar et al 2020) were also proposed for HMER. Although such methods can explicitly parse the primitives corresponding to the target token, they require complicated manual work to design grammars, and thus, are less efficient compared with data driven learning.…”

Improving Semantic Segmentation with Graph-Based Structural Knowledge

“…This type of high-level structural information is very promising [2,8,9,19,6] and it has found applications in medical image understanding [4,7,18] but also in document analysis (e.g. [5,12] for handwriting recognition) or in scene understanding (e.g. [13] for robotic).…”

Arrow R-CNN for handwritten diagram recognition

“…This input device captures the drawing as a temporal sequence of strokes. Online diagram recognition has received a lot of attention in research, especially in the area of flowcharts [1][2][3][4][5][6]9,14,18,36,37,40]. Yet, those approaches are of limited applicability if the original stroke data are not available (e.g., hand-drawn diagrams on paper).…”