Finding the Evidence: Localization-aware Answer Prediction for Text Visual Question Answering

Han, Wei; Huang, Hantao; Han, Tao

doi:10.18653/v1/2020.coling-main.278

Cited by 40 publications

(15 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…TextVQA attracts more and more attention from communities since it focuses on texts in natural daily scenes, such as road signs and displays. To promote progress in this field, several datasets [4,11,12] and methods [8,4,13,5] have been proposed. LoRRA [4] is the pioneering model which utilizes an attention mechanism to handle image features, OCR features, and question words, but it can only select one word as the answer.…”

Section: Text-based Visual Question Answeringmentioning

confidence: 99%

“…The Multimodal Multi-Copy Mesh (M4C) model [5] boosts TextVQA performance by employing a multimodal transformer [6] to fuse various modality entities. In the following works, modifications with respect to feature embedding [9], feature interaction [14,7,15] and answer decoding [8] have been shown to improve performance. In addition, a pre-training method TAP [10] is proposed on the TextVQA and TextCaption tasks.…”

Section: Text-based Visual Question Answeringmentioning

confidence: 99%

“…M4C [5] enriches OCR representations and uti- lizes a multimodal transformer [6] to project all entities from each modality (questions, OCR tokens and visual objects) to a joint embedding space. Although notable improvements have been achieved with recent promising techniques [7,8,9,10], many challenges still exist. Generally, the answers to the text-based questions are sequentially selected from a fixed vocabulary that consists of high-frequency words in training set and a dynamic vocabulary composed of OCR result tokens in the image.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Towards Escaping from Language Bias and OCR Error: Semantics-Centered Text Visual Question Answering

Fang¹,

Zeng²,

Zhou³

et al. 2022

Preprint

View full text Add to dashboard Cite

Texts in scene images convey critical information for scene understanding and reasoning. The abilities of reading and reasoning matter for the model in the text-based visual question answering (TextVQA) process. However, current TextVQA models do not center on the text and suffer from several limitations. The model is easily dominated by language biases and optical character recognition (OCR) errors due to the absence of semantic guidance in the answer prediction process. In this paper, we propose a novel Semantics-Centered Network (SC-Net) that consists of an instance-level contrastive semantic prediction module (ICSP) and a semantics-centered transformer module (SCT). Equipped with the two modules, the semantics-centered model can resist the language biases and the accumulated errors from OCR. Extensive experiments on TextVQA and ST-VQA datasets show the effectiveness of our model. SC-Net surpasses previous works with a noticeable margin and is more reasonable for the TextVQA task.

show abstract

Section: Text-based Visual Question Answeringmentioning

confidence: 99%

Section: Text-based Visual Question Answeringmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Towards Escaping from Language Bias and OCR Error: Semantics-Centered Text Visual Question Answering

Fang¹,

Zeng²,

Zhou³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This line of research has been pursued by several studies, particularly thanks to the introduction of the Fact‐based VQA dataset (Wang et al., 2017b). The VQA task is now taking new directions, such as embodied approaches where an agent has to navigate an environment and answer questions about it (H. Chen et al., 2019; Das et al., 2018); video VQA, where the answer has to be found in videos rather than in static images (Lei et al., 2018, 2020); answering questions about diagrams and charts (Ebrahimi Kahou et al., 2017; Kafle et al., 2018); text VQA, which involves recognizing and interpreting textual content in images (Biten et al., 2019; Han et al., 2020); answering questions about medical images (see, Abacha et al., 2020); and many others.…”

Section: The Recent Revival Of Vqamentioning

confidence: 99%

Linguistic issues behind visual question answering

Bernardi

Pezzelle

2021

Language and Linguist. Compass

View full text Add to dashboard Cite

Answering a question that is grounded in an image is a crucial ability that requires understanding the question, the visual context, and their interaction at many linguistic levels: among others, semantics, syntax and pragmatics. As such, visually‐grounded questions have long been of interest to theoretical linguists and cognitive scientists. Moreover, they have inspired the first attempts to computationally model natural language understanding, where pioneering systems were faced with the highly challenging task—still unsolved—of jointly dealing with syntax, semantics and inference whilst understanding a visual context. Boosted by impressive advancements in machine learning, the task of answering visually‐grounded questions has experienced a renewed interest in recent years, to the point of becoming a research sub‐field at the intersection of computational linguistics and computer vision. In this paper, we review current approaches to the problem which encompass the development of datasets, models and frameworks. We conduct our investigation from the perspective of the theoretical linguists; we extract from pioneering computational linguistic work a list of desiderata that we use to review current computational achievements. We acknowledge that impressive progress has been made to reconcile the engineering with the theoretical view. At the same time, we claim that further research is needed to get to a unified approach which jointly encompasses all the underlying linguistic problems. We conclude the paper by sharing our own desiderata for the future.

show abstract

“…On TextCaps, M4C can be adapted to generate a sentence by taking previously generated words as text inputs at each time step. Multiple models have been introduced recently which ablate various components of M4C for better accuracy [24,13,16,22]. Contrary to M4C and derivative works which treat OCR as a black box, in PixelM4C, we train an end-to-end model and use this capability to apply new design choices in a more informed way.…”

Section: Downstream Application Modelsmentioning

confidence: 99%

TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text

Singh¹,

Pang²,

Toh³

et al. 2021

Preprint

View full text Add to dashboard Cite

A crucial component for the scene text based reasoning required for TextVQA and TextCaps datasets involve detecting and recognizing text present in the images using an optical character recognition (OCR) system. The current systems are crippled by the unavailability of ground truth text annotations for these datasets as well as lack of scene text detection and recognition datasets on real images disallowing the progress in the field of OCR and evaluation of scene text based reasoning in isolation from OCR systems. In this work, we propose TextOCR, an arbitrary-shaped scene text detection and recognition with 900k annotated words collected on real images from TextVQA dataset. We show that current state-of-the-art text-recognition (OCR) models fail to perform well on TextOCR and that training on TextOCR helps achieve state-of-the-art performance on multiple other OCR datasets as well. We use a TextOCR trained OCR model to create PixelM4C model which can do scene text based reasoning on an image in an end-to-end fashion, allowing us to revisit several design choices to achieve new state-of-the-art performance on TextVQA dataset.

show abstract

Finding the Evidence: Localization-aware Answer Prediction for Text Visual Question Answering

Cited by 40 publications

References 16 publications

Towards Escaping from Language Bias and OCR Error: Semantics-Centered Text Visual Question Answering

Towards Escaping from Language Bias and OCR Error: Semantics-Centered Text Visual Question Answering

Linguistic issues behind visual question answering

TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text

Contact Info

Product

Resources

About