Pure Transformer with Integrated Experts for Scene Text Recognition

Tan, Yew Lee; Kong, Adams Wai-Kin; Kim, Jung‐Jae

doi:10.1007/978-3-031-19815-1_28

Cited by 7 publications

(5 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…, SRN [8], Vision-LAN [55], PARSeq [57], ABINet++ [104], LevOCR [56], MA-TRN [10] and MGP-STR) perform better than language-free methods, showing the significance of linguistic information. PTIE [51], which utilizes a transformer-only model with multiple patch resolutions, also achieves good results. Notably, owing to the multi-granularity predictions, MGP-STR † CF S has already outperformed the recent stateof-the-art method MATRN.…”

Section: Results On Standard Benchmarksmentioning

confidence: 95%

“…Inspired by the great success of Transformer [49] in natural language processing (NLP) tasks, the application of Transformer in STR has also attracted more attention [50], [51], [52]. Vision Transformer (ViT) [11] that directly processes image patches without convolutions opens the beginning of using Transformer blocks instead of CNNs to solve computer vision problems [53], [54], leading to prominent results.…”

Section: Language-free Str Methodsmentioning

confidence: 99%

“…ViTSTR [50] attempts to simply leverage the feature representations of the last layer of ViT for parallel character decoding. PTIE [51] is a transformer model that can process multiple patch resolutions and decode in both the original and reverse character orders. In general, languagefree methods often fail to recognize low-quality images due to the lack of language information.…”

Section: Language-free Str Methodsmentioning

confidence: 99%

See 2 more Smart Citations

Multi-granularity Prediction for Scene Text Recognition

Wang

Cheng

Yao

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Due to the enormous technical challenges and wide range of applications, scene text recognition (STR) has been an active research topic in computer vision for years. To tackle this tough problem, numerous innovative methods have been successively proposed, and incorporating linguistic knowledge into STR models has recently become a prominent trend. In this work, we first draw inspiration from the recent progress in Vision Transformer (ViT) to construct a conceptually simple yet functionally powerful vision STR model, which is built upon ViT and a tailored Adaptive Addressing and Aggregation (A 3 ) module. It already outperforms most previous state-of-the-art models for scene text recognition, including both pure vision models and language-augmented methods. To integrate linguistic knowledge, we further propose a Multi-Granularity Prediction strategy to inject information from the language modality into the model in an implicit way, i.e. , subword representations (BPE and WordPiece) widely used in NLP are introduced into the output space, in addition to the conventional character level representation, while no independent language model (LM) is adopted. To produce the final recognition results, two strategies for effectively fusing the multi-granularity predictions are devised: the first strategy, called Confidence-based Fusion Strategy (CFS), employs a straightforward rule based on the confidence scores of the multi-granularity predictions, while the second one, called Learnable Fusion Strategy (LFS), is realized with a trainable module to directly measure the cross-modal similarities between text (predicted words) and images, akin to CLIP. The resultant algorithm (termed MGP-STR) is able to push the performance envelope of STR to an even higher level. Specifically, MGP-STR with LFS achieves an average recognition accuracy of 94% on standard benchmarks for scene text recognition (such as IIIT 5K-word, ICDAR 2015, SVT and CUTE). Moreover, it also achieves state-of-the-art results on widely-used handwritten benchmarks (IAM, CVL and RIMES) as well as more challenging scene text datasets (ArT, COCO-Text and Uber-Text), demonstrating the generality of the proposed MGP-STR algorithm. The source code and models will be available at: https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/OCR/MGP-STR.

show abstract

Section: Results On Standard Benchmarksmentioning

confidence: 95%

Section: Language-free Str Methodsmentioning

confidence: 99%

Section: Language-free Str Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Multi-granularity Prediction for Scene Text Recognition

Wang

Cheng

Yao

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…However, the above approaches utilize a fixed patch resolution when segmenting with ViT, which can negatively impact text images of specific word lengths and scaling scales, [16] proposed PTIE (Pure Transformer with Integrated Experts), a pure Transformer model that accommodates different patch resolutions and decodes in both the original and reverse character orders. MRP-STR [17] presented a custom Adaptive Addressing and Aggregation (A3) module that selects meaningful token combinations from ViT and integrates them into an output token corresponding to a specific character, named the Character A3 module.…”

Section: Transformer Based Methodsmentioning

confidence: 99%

RMLP-ViT: Recognizing Handwritten Chinese Examination Texts with ViT

Zhong,

Zhu

2024

Preprint

View full text Add to dashboard Cite

It happens in the examination paper that text lines include inconsistent nonuniform word size, character erasure, diverse text length and dense long texts. This paper proposes an improved method for ViT to enhance its capability in recognizing text lines in handwritten Chinese examination papers. First, this method employs a segmentation method suitable for text line recognition and proposes a repeated multiscale linear projection (RMLP) method to enrich the spatial information of the image vectors, which improves the model's integration capability for patch vectors of multiple scales. Second, ViT is combined with CTC to achieve prediction for each patch, thus improving the robustness of ViT in Chinese handwritten text recognition. Experiments show that RMLP-ViT promises the recognition of examination paper text lines and achieves good performance on the SCUT-EPT dataset.

show abstract

“…Biten et al [27] propose a layer-aware transformer with a pre-training scheme on the basis of text and spatial cues only and show that it works well on scanned documents to handle multimodality in scene text visual question answering. Based on ViT [4], Tan et al [28] propose a mixture experts of pure transformers for processing different resolutions for scene text recognition. Atienza [5] presents a new transformer, called ViTSTR, for scent text recognition, which only uses the encoder architecture and emphasizes the balance between the performance and computational efficiency.…”

Section: B Vision Transformers For Strmentioning

confidence: 99%

Augmented Transformers with Adaptive n-grams Embedding for Multilingual Scene Text Recognition

Yan¹,

Fang²,

Jin³

2023

Preprint

View full text Add to dashboard Cite

While vision transformers have been highly successful in improving the performance in image-based tasks, not much work has been reported on applying transformers to multilingual scene text recognition due to the complexities in the visual appearance of multilingual texts. To fill the gap, this paper proposes an augmented transformer architecture with ngrams embedding and cross-language rectification (TANGER). TANGER consists of a primary transformer with single patch embeddings of visual images, and a supplementary transformer with adaptive n-grams embeddings that aims to flexibly explore the potential correlations between neighbouring visual patches, which is essential for feature extraction from multilingual scene texts. Cross-language rectification is achieved with a loss function that takes into account both language identification and contextual coherence scoring. Extensive comparative studies are conducted on four widely used benchmark datasets as well as a new multilingual scene text dataset containing Indonesian, English, and Chinese collected from tourism scenes in Indonesia. Our experimental results demonstrate that TANGER is considerably better compared to the state-of-the-art, especially in handling complex multilingual scene texts.

show abstract

Pure Transformer with Integrated Experts for Scene Text Recognition

Cited by 7 publications

References 44 publications

Multi-granularity Prediction for Scene Text Recognition

Multi-granularity Prediction for Scene Text Recognition

RMLP-ViT: Recognizing Handwritten Chinese Examination Texts with ViT

Augmented Transformers with Adaptive n-grams Embedding for Multilingual Scene Text Recognition

Contact Info

Product

Resources

About