Visual Semantics Allow for Textual Reasoning Better in Scene Text Recognition

Abstract

Existing Scene Text Recognition (STR) methods typically use a language model to optimize the joint probability of the 1D character sequence predicted by a visual recognition (VR) model, which ignore the 2D spatial context of visual semantics within and between character instances, making them not generalize well to arbitrary shape scene text. To address this issue, we make the first attempt to perform textual reasoning based on visual semantics in this paper. Technically, given the character segmentation maps predicted by a VR model, we construct a subgraph for each instance, where nodes represent the pixels in it and edges are added between nodes based on their spatial similarity. Then, these subgraphs are sequentially connected by their root nodes and merged into a complete graph. Based on this graph, we devise a graph convolutional network for textual reasoning (GTR) by supervising it with a cross-entropy loss. GTR can be easily plugged in representative STR models to improve their performance owing to better textual reasoning. Specifically, we construct our model, namely S-GTR, by paralleling GTR to the language model in a segmentation-based STR baseline, which can effectively exploit the visual-linguistic complementarity via mutual learning. S-GTR sets new state-of-the-art on six challenging STR benchmarks and generalizes well to multi-linguistic datasets. Code is available at https://github.com/adeline-cs/GTR.

A survey on methods, datasets and implementations for scene text spotting

“…[95] proposed a multi‐stage and multi‐scale decoder that jointly carries out visual and semantic reasoning, with recurrent stages refining the results. Lastly, [96] proposes a graph‐based solution that groups pixels in a character instance based on their location similarity, achieving state‐of‐the‐art results.…”

“…The performance of the recognition methods can be seen in Table 4. The most prominent method in ICDAR 2013 dataset is S‐GTR [96] with its graph‐based convolutional network and visual semantic knowledge, obtaining an accuracy of 97.80%. On ICDAR 2015, this method was also the most successful approach, obtaining a score of 87.30%.…”

Masked Vision-Language Transformers for Scene Text Recognition

“…VST [35] extracts semantics from a visual feature map and performs a visual-semantic interaction using an alignment module. Most recently, some language-aware models explore more possibilities to boost the performance by considering spatial context [13], adding real-world images to train the model in a semi-supervised way [1], or developing a re-ranking method to get a better candidate output [30], while our work takes a step by extracting both the explicit and the implicit semantics.…”

NRSTRNet: A Novel Network for Noise-Robust Scene Text Recognition

“…Unlike traditional text recognition methods, deep learning methods [5][6][7][8][9][10][11] perform a top-down approach to recognize words or text lines directly, showing more promising results. Recent approaches [4,12] treat the STR problem as a sequence prediction problem, resulting in increased precision while eliminating the need for character-level annotation.…”