Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes

Abstract

Unifying text detection and text recognition in an end-to-end training fashion has become a new trend for reading text in the wild, as these two tasks are highly relevant and complementary. In this paper, we investigate the problem of scene text spotting, which aims at simultaneous text detection and recognition in natural images. An end-to-end trainable neural network named as Mask TextSpotter is presented. Different from the previous text spotters that follow the pipeline consisting of a proposal generation network and a sequence-to-sequence recognition network, Mask TextSpotter enjoys a simple and smooth end-to-end learning procedure, in which both detection and recognition can be achieved directly from two-dimensional space via semantic segmentation. Further, a spatial attention module is proposed to enhance the performance and universality. Benefiting from the proposed two-dimensional representation on both detection and recognition, it easily handles text instances of irregular shapes, for instance, curved text. We evaluate it on four English datasets and one multi-language dataset, achieving consistently superior performance over state-of-the-art methods in both detection and end-to-end text recognition tasks. Moreover, we further investigate the recognition module of our method separately, which significantly outperforms state-of-the-art methods on both regular and irregular text datasets for scene text recognition.

Toward Arbitrary-Shaped Text Spotting Based on End-to-End

“…Compared with [28], our method showed an improvement of 1.2% and 3.7% in text detection tasks and the end-to-end text spotting with strong lexicon, respectively. In addition, compared with [27], our method improves text detection performance by 2.8%.…”

“…The results are shown in Table IV. Compared with [27], our method improves 1.2% and 0.9% in text detection tasks and end-to-end text spotting tasks, respectively. "P", "R" and "F" mean Precision, Recall and F-measure in detection task respectively.…”

“…In particular, compared with the method in [28], the performance of the method in this paper is 0.9% and 2.4% higher than that of Boundary in text detection tasks and end-to-end text spotting (without lexicon) tasks respectively. Compared with [27], it improved by 3.0% in text detection tasks. It should be noted that [27] requires characterlevel annotations.…”

“…Compared with [27], it improved by 3.0% in text detection tasks. It should be noted that [27] requires characterlevel annotations. The reasons for the performance improvement are as follows, first, the corner attention mechanism is conducive to the detection and recognition of long text; second, the text decoder based on SA-BiLSTM can better extract the semantic information of the text; finally, TPDM can better avoid interference from complex background.…”

“…SAM is proposed in the paper [27]. In contrast, we combine the spatial attention mechanism with BiLSTM to better extract the semantic information between texts.…”