Masked Vision-Language Transformers for Scene Text Recognition

Abstract

Scene text recognition (STR) enables computers to recognize and read the text in various real-world scenes. Recent STR models benefit from taking linguistic information in addition to visual cues into consideration. We propose a novel Masked Vision-Language Transformers (MVLT) to capture both the explicit and the implicit linguistic information. Our encoder is a Vision Transformer, and our decoder is a multi-modal Transformer. MVLT is trained in two stages: in the first stage, we design a STR-tailored pretraining method based on a masking strategy; in the second stage, we fine-tune our model and adopt an iterative correction method to improve the performance. MVLT attains superior results compared to state-of-the-art STR models on several benchmarks. Our code and model are available at https://github.com/onealwj/MVLT.

PTF-Vāc:Ab-initiodiscovery of plant transcription factors binding sites using explainable and generative deep co-learning encoders-decoders

“…Subsequently, the final output undergoes conditional probability estimation, unfolding as a crucial aspect of the decoding mechanism. Building upon the output after layer normalization, the model proceeds to compute the conditional probability distribution over the vocabulary for the subsequent token 32 . This involves employing the softmax function, transforming raw scores into a probability distribution.…”