Visual Semantics Allow for Textual Reasoning Better in Scene Text Recognition

Abstract

Existing Scene Text Recognition (STR) methods typically use a language model to optimize the joint probability of the 1D character sequence predicted by a visual recognition (VR) model, which ignore the 2D spatial context of visual semantics within and between character instances, making them not generalize well to arbitrary shape scene text. To address this issue, we make the first attempt to perform textual reasoning based on visual semantics in this paper. Technically, given the character segmentation maps predicted by a VR model, we construct a subgraph for each instance, where nodes represent the pixels in it and edges are added between nodes based on their spatial similarity. Then, these subgraphs are sequentially connected by their root nodes and merged into a complete graph. Based on this graph, we devise a graph convolutional network for textual reasoning (GTR) by supervising it with a cross-entropy loss. GTR can be easily plugged in representative STR models to improve their performance owing to better textual reasoning. Specifically, we construct our model, namely S-GTR, by paralleling GTR to the language model in a segmentation-based STR baseline, which can effectively exploit the visual-linguistic complementarity via mutual learning. S-GTR sets new state-of-the-art on six challenging STR benchmarks and generalizes well to multi-linguistic datasets.

Automated hand-marked semantic text recognition from photographs

“…This study makes a significant contribution by addressing the unexplored but practical HMSTR task, which cannot be effectively handled by state-of-the-art OCR and STR techniques 13 . Specifically, our experiments demonstrate that the proposed approach effectively recognizes hand-marked domain-specific terms, jargon, loan words, and abbreviations 28 , for which other typical text recognition models lack training.…”

“…Recent advancements in STR have focused on developing context-aware methods to improve text recognition accuracy. For instance, He et al 13 proposed a segmentation baseline with graph-based textual reasoning (S-GTR), which utilizes a graph convolutional network to incorporate visual context for textual reasoning. Bautista and Atienza 14 enhanced STR performance by combining context-free and context-aware autoregressive inference.…”

“…However, even a state-of-the-art STR model 13 struggles with HMSTR owing to the presence of irregular hand-drawn markers, adding complexity to the localization and recognition of the text. This difficulty arises from several factors.…”

OCR-IDL: OCR Annotations for Industry Document Library Dataset

“…More specifically, they obtain OCR annotations from an open source OCR engine Tesseract [45] for 5 Million documents from IIT-CDIP [25] dataset. With the introduction of pre-training strategy and advances in modern OCR engine [1,12,20,28,34], many contemporary approaches [7,2,53] have utilized even more data to advance the Document Intelligence field.…”