Towards VQA Models That Can Read

Abstract

Studies have shown that a dominant class of questions asked by visually impaired users on images of their surroundings involves reading text in the image. But today's VQA models can not read! Our paper takes a first step towards addressing this problem. First, we introduce a new "TextVQA" dataset to facilitate progress on this important problem. Existing datasets either have a small proportion of questions about text (e.g., the VQA dataset) or are too small (e.g., the VizWiz dataset). TextVQA contains 45,336 questions on 28,408 images that require reasoning about text to answer. Second, we introduce a novel model architecture that reads text in the image, reasons about it in the context of the image and the question, and predicts an answer which might be a deduction based on the text and the image or is composed of the strings found in the image. Consequently, we call our approach Look, Read, Reason & Answer (LoRRA) 1 . We show that LoRRA outperforms existing state-of-the-art VQA models on our TextVQA dataset. We find that the gap between human performance and machine performance is significantly larger on TextVQA than on VQA 2.0, suggesting that TextVQA is well-suited to benchmark progress along directions complementary to VQA 2.0. VQA ComponentSimilar to many VQA models [7,17], we first embed the question words w 1 , w 2 , . . . , w L of the question q with a pre-trained embedding function (e.g. GloVe [36]) and then encode the resultant word embeddings iteratively with a re-

ICDAR 2019 Competition on Scene Text Visual Question Answering

“…The ST-VQA Challenge ran between February and April 2019. Participants were provided with a training set at the beginning of March, while the test set images and questions were only made available for a two week period between [15][16][17][18][19][20][21][22][23][24][25][26][27][28][29][30] April. The participants were requested to submit results over the test set images and not executables of their systems.…”

“…Interestingly, concurrently with the ST-VQA challenge, a work similar to ours introduced a new dataset [24] called Text-VQA. This work and the corresponding dataset were published while ST-VQA challenge was on-going.…”

Improving Intra- and Inter-Modality Visual Relation for Image Captioning

“…Since intra-modality features can be seen as the result of sampling from the distribution along each channel, similarity scores computed over fixed distribution depict feature interactions more profoundly. Moreover, we consider that detector-based features [2,29] may fail to cover all object details, which restricts performance of captioning. Consequently, we further recommend fusing detector-based and grid-based [29] features in image encoder, which helps to enrich object representations.…”

“…Moreover, we consider that detector-based features [2,29] may fail to cover all object details, which restricts performance of captioning. Consequently, we further recommend fusing detector-based and grid-based [29] features in image encoder, which helps to enrich object representations. By combining both CW Norm and multi-level features, we construct our Relation Enhanced Transformer Block (RETB) for image feature learning.…”

Look, Read and Feel: Benchmarking Ads Understanding with Multimodal Multitask Learning

“…To our best knowledge, this is the first framework that unifies the topic and sentiment understanding of ads. In particular, we first extract different types of information, such as objects and contained texts from ads using some existing techniques, such as the pre-trained object or image representation models and OCR [29,30]. To recognize and understand the visual rhetoric, an autoencoder module is introduced to decode the object representation in an unsupervised manner.…”