Textual Description for Mathematical Equations

Abstract

Reading of mathematical expression or equation in the document images is very challenging due to the large variability of mathematical symbols and expressions. In this paper, we pose reading of mathematical equation as a task of generation of the textual description which interprets the internal meaning of this equation. Inspired by the natural image captioning problem in computer vision, we present a mathematical equation description (MED) model, a novel end-to-end trainable deep neural network based approach that learns to generate a textual description for reading mathematical equation images. Our MED model consists of a convolution neural network as an encoder that extracts features of input mathematical equation images and a recurrent neural network with attention mechanism which generates description related to the input mathematical equation images. Due to the unavailability of mathematical equation image data sets with their textual descriptions, we generate two data sets for experimental purpose. To validate the effectiveness of our MED model, we conduct a real-world experiment to see whether the students are able to write equations by only reading or listening their textual descriptions or not. Experiments conclude that the students are able to write most of the equations correctly by reading their textual descriptions only.

Automatic Adaptation of Open Educational Resources: An Approach From a Multilevel Methodology Based on Students’ Preferences, Educational Special Needs, Artificial Intelligence and Accessibility Metadata

“…In the case of photographs, the neural network architecture is based on [35] and equations based on [36]. This architecture is based on the combination of CNN-RNN.…”

“…In the description of the photographs, the MS COCO dataset database was used, which has 180k training images with their respective annotations; as an addition to this database, 2k images with their description referring to educational topics were included. In the case of equations description, we made use of a database of 400k images with their annotations, this database was generated according to [36], as an aggregate we expanded characters and equations.…”