Don’t Forget Me: Accurate Background Recovery for Text Removal via Modeling Local-Global Context

Liu, Chongyu; Jin, Lianwen; Liu, Yuliang; Luo, Canjie; Chen, Bangdong; Guo, Fengjun; Ding, Kai

doi:10.1007/978-3-031-19815-1_24

Cited by 9 publications

(3 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Following previous studies (Liu et al 2020(Liu et al , 2022a, the image-eval metrics include PSNR, MSSIM, MSE, AGE, pEPs, pCEPs, and FID, while the detection-eval metrics involve the precision (P), recall (R), and f-measure (F) using the pretrained CRAFT (Baek et al 2019) for text detection.…”

Section: Evaluation Metricsmentioning

confidence: 99%

“…The sequential text localizing and background inpainting pipeline introduces additional parameters, decreases the inference speed, and, more importantly, breaks the integrity of the entire model. The error of text localization can be easily propagated to the background inpainting, especially for the methods that require pre-supplied text detectors (Tang et al 2021;Liu et al 2022a; Lee and Choi 2022). (2) Recent advances (Liu et al 2020;Lyu and Zhu 2022;Du et al 2023b;Wang et al 2023) tend to employ a multi-step paradigm in a coarse-to-fine or progressive fashion, which significantly undermines efficiency and makes it difficult to balance the parameters involved in multiple steps.…”

Section: Introductionmentioning

confidence: 99%

“…The two-stage methods can be further divided into separate and end-to-end categories. The separate two-stage methods depend on separately trained text detectors (Zdenek and Nakayama 2020;Conrad and Chen 2021;Liu et al 2022a) or ground truth (GT) (Qin, Wei, and Manduchi 2018;Tursun et al 2019b;Tang et al 2021;Lee and Choi 2022) to obtain text masks. In contrast, end-to-end two-stage methods end-to-end optimize the text localizing modules with other components (Keserwani and Roy 2021).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

ViTEraser: Harnessing the Power of Vision Transformers for Scene Text Removal with SegMIM Pretraining

Peng,

Liu,

Liu

et al. 2024

AAAI

View full text Add to dashboard Cite

Scene text removal (STR) aims at replacing text strokes in natural scenes with visually coherent backgrounds. Recent STR approaches rely on iterative refinements or explicit text masks, resulting in high complexity and sensitivity to the accuracy of text localization. Moreover, most existing STR methods adopt convolutional architectures while the potential of vision Transformers (ViTs) remains largely unexplored. In this paper, we propose a simple-yet-effective ViT-based text eraser, dubbed ViTEraser. Following a concise encoder-decoder framework, ViTEraser can easily incorporate various ViTs to enhance long-range modeling. Specifically, the encoder hierarchically maps the input image into the hidden space through ViT blocks and patch embedding layers, while the decoder gradually upsamples the hidden features to the text-erased image with ViT blocks and patch splitting layers. As ViTEraser implicitly integrates text localization and inpainting, we propose a novel end-to-end pretraining method, termed SegMIM, which focuses the encoder and decoder on the text box segmentation and masked image modeling tasks, respectively. Experimental results demonstrate that ViTEraser with SegMIM achieves state-of-the-art performance on STR by a substantial margin and exhibits strong generalization ability when extended to other tasks, e.g., tampered scene text detection. Furthermore, we comprehensively explore the architecture, pretraining, and scalability of the ViT-based encoder-decoder for STR, which provides deep insights into the application of ViT to the STR field. Code is available at https://github.com/shannanyinxiang/ViTEraser.

show abstract

Section: Evaluation Metricsmentioning

confidence: 99%