Mask TextSpotter v3: Segmentation Proposal Network for Robust Scene Text Spotting

Liao, Minghui; Pang, Guangchang; Huang, Jing; Hassner, Tal; Bai, Xiang

doi:10.1007/978-3-030-58621-8_41

Cited by 164 publications

(138 citation statements)

References 43 publications

Supporting

Mentioning

138

Contrasting

Order By: Relevance

“…The backbone of Mask TextSpotter V3 is ResNet-50 (He et al, 2016). Proposals are generated by an SPN (Table 1) (Liao et al, 2020), as opposed to the Mask TextSpotter V2 RPN). The SPN provides polygonal (rather than rectangular) representations for the proposals, meaning that the curving or other unaligned features are better detected.…”

Section: Methodsmentioning

confidence: 99%

“…This function modifies the image and label based on a random counterclockwise rotation angle between 0 and 90 °, including 0 but not 90. This range was chosen to reflect the same range for rotation augmentations in the MTS V3 paper (Liao et al, 2020).…”

Section: Methodsmentioning

confidence: 99%

“…However, end-toend code repositories such as TextFuseNet (Ye et al, 2020), which uses a multi-path feature fusion architecture, and Mask TextSpotter (MTS), which is an end-to-end PyTorch code repository, have recently emerged that tie the two tasks together. For this study, we used MTS, which implements a trainable neural network for spotting text with arbitrary shapes (Lyu et al, 2018;Liao et al, 2020). MTS is based on Mask R-CNN, which extends Faster R-CNN by adding a branch that computes the object mask prediction in parallel with the existing branch for bounding box recognition (He et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Deep Learning Detection and Recognition of Spot Elevations on Historical Topographic Maps

Arundel

Morgan

Thiem

2022

Front. Environ. Sci.

View full text Add to dashboard Cite

Some information contained in historical topographic maps has yet to be captured digitally, which limits the ability to automatically query such data. For example, U.S. Geological Survey’s historical topographic map collection (HTMC) displays millions of spot elevations at locations that were carefully chosen to best represent the terrain at the time. Although research has attempted to reproduce these data points, it has proven inadequate to automatically detect and recognize spot elevations in the HTMC. We propose a deep learning workflow pretrained using large benchmark text datasets. To these datasets we add manually crafted training image/label pairs, and test how many are required to improve prediction accuracy. We find that the initial model, pretrained solely with benchmark data, fails to predict any HTMC spot elevations correctly, whereas the addition of just 50 custom image/label pairs increases the predictive ability by ∼50%, and the inclusion of 350 data pairs increased performance by ∼80%. Data augmentation in the form of rotation, scaling, and translation (offset) expanded the size and diversity of the training dataset and vastly improved recognition accuracy up to ∼95%. Visualization methods, such as heat map generation and salient feature detection, can be used to better understand why some predictions fail.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Deep Learning Detection and Recognition of Spot Elevations on Historical Topographic Maps

Arundel

Morgan

Thiem

2022

Front. Environ. Sci.

View full text Add to dashboard Cite

show abstract

“…Typical structure extraction methods rely on preliminary Optical Character Recognition (OCR) engines [19,34,39,40,47,49] to understand the semantics of documents. As shown in Figure 1, the contents in a document can be located as several text segments (pink dotted boxes) by text detectors.…”

Section: Introductionmentioning

confidence: 99%

StrucTexT: Structured Text Understanding with Multi-Modal Transformers

Qian

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Structured text understanding on Visually Rich Documents (VRDs) is a crucial part of Document Intelligence. Due to the complexity of content and layout in VRDs, structured text understanding has been a challenging task. Most existing studies decoupled this problem into two sub-tasks: entity labeling and entity linking, which require an entire understanding of the context of documents at both token and segment levels. However, little work has been concerned with the solutions that efficiently extract the structured data from different levels. This paper proposes a unified framework named StrucTexT, which is flexible and effective for handling both sub-tasks. Specifically, based on the transformer, we introduce a segment-token aligned encoder to deal with the entity labeling and entity linking tasks at different levels of granularity. Moreover, we design a novel pre-training strategy with three self-supervised tasks to learn a richer representation. StrucTexT uses the existing Masked Visual Language Modeling task and the new Sentence Length Prediction and Paired Boxes Direction tasks to incorporate the multi-modal information across text, image, and layout. We evaluate our method for structured text understanding at segment-level and token-level and show it outperforms the state-of-the-art counterparts with significantly superior performance on the FUNSD, SROIE, and EPHOIE datasets.

show abstract

“…Scene text spotting has witnessed remarkable progress and achieved promising results (Liu et al 2018;Lyu et al 2018;Liu et al 2020;Liao et al 2020) in recent years. However, there is still room for improvement in scene text spotting, due to the inconsistency between text detection and recognition, which involves the following two aspects.…”

Section: Introductionmentioning

confidence: 99%

ARTS: Eliminating Inconsistency between Text Detection and Recognition with Auto-Rectification Text Spotter

Zhong¹,

Tang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent approaches for end-to-end text spotting have achieved promising results. However, most of the current spotters were plagued by the inconsistency problem between text detection and recognition. In this work, we introduce and prove the existence of the inconsistency problem and analyze it from two aspects: (1) inconsistency of text recognition features between training and testing, and (2) inconsistency of optimization targets between text detection and recognition. To solve the aforementioned issues, we propose a differentiable Auto-Rectification Module (ARM) together with a new training strategy to enable propagating recognition loss back into detection branch, so that our detection branch can be jointly optimized by detection and recognition targets, which largely alleviates the inconsistency problem between text detection and recognition. Based on these designs, we present a simple yet robust end-to-end text spotting framework, termed Auto-Rectification Text Spotter (ARTS), to detect and recognize arbitrarily-shaped text in natural scenes. Extensive experiments demonstrate the superiority of our method. In particular, our ARTS-S achieves 77.1% end-to-end text spotting F-measure on Total-Text at a competitive speed of 10.5 FPS, which significantly outperforms previous methods in both accuracy and inference speed.

show abstract

Mask TextSpotter v3: Segmentation Proposal Network for Robust Scene Text Spotting

Cited by 164 publications

References 43 publications

Deep Learning Detection and Recognition of Spot Elevations on Historical Topographic Maps

Deep Learning Detection and Recognition of Spot Elevations on Historical Topographic Maps

StrucTexT: Structured Text Understanding with Multi-Modal Transformers

ARTS: Eliminating Inconsistency between Text Detection and Recognition with Auto-Rectification Text Spotter

Contact Info

Product

Resources

About