Overcoming Language Priors in VQA via Decomposed Linguistic Representations

Jing, Chenchen; Wu, Yuwei; Zhang, Xiaoxun; Jia, Yingmin; Wu, Qi

doi:10.1609/aaai.v34i07.6776

Cited by 73 publications

(38 citation statements)

References 24 publications

(42 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast, federated learning is used with the aimNet and is validated on federated learning settings that include both horizontal and vertical federated learning. To focus on language priors, a modular language attention mechanism is used by Jing et al (2020) to parse a question into three phrase representations, namely type representation, object representation, and concept representation. It has prevented language priors from dominating the answering process.…”

Section: Image Question Answering Modelmentioning

confidence: 99%

Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods

Mogadala

Kalimuthu

Klakow

2021

jair

View full text Add to dashboard Cite

Interest in Artificial Intelligence (AI) and its applications has seen unprecedented growth in the last few years. This success can be partly attributed to the advancements made in the sub-fields of AI such as machine learning, computer vision, and natural language processing. Much of the growth in these fields has been made possible with deep learning, a sub-area of machine learning that uses artificial neural networks. This has created significant interest in the integration of vision and language. In this survey, we focus on ten prominent tasks that integrate language and vision by discussing their problem formulation, methods, existing datasets, evaluation measures, and compare the results obtained with corresponding state-of-the-art methods. Our efforts go beyond earlier surveys which are either task-specific or concentrate only on one type of visual content, i.e., image or video. Furthermore, we also provide some potential future directions in this field of research with an anticipation that this survey stimulates innovative thoughts and ideas to address the existing challenges and build new applications.

show abstract

Section: Image Question Answering Modelmentioning

confidence: 99%

Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods

Mogadala

Kalimuthu

Klakow

2021

jair

View full text Add to dashboard Cite

show abstract

“…3) The data rebalance-based methods [9,15,28,31,51] attempt to propose some data augmentation strategies to generate counterfactual training instances automatically, thereby balancing the answer distribution of training data. 4) The other methods: there are also many impressive works to overcome language bias through adversarial learning [37], modifying language module [21,27], and casual inference [33]. Among most of these debiasing methods [8,12,33,37], the question-only (unimodal) branch is crucial for capturing spurious relationships between questions and answer candidates.…”

Section: Related Workmentioning

confidence: 99%

From Superficial to Deep: Language Bias driven Curriculum Learning for Visual Question Answering

Lao

Guo

Liu

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Most Visual Question Answering (VQA) models are faced with language bias when learning to answer a given question, thereby failing to understand multimodal knowledge simultaneously. Based on the fact that VQA samples with dierent levels of language bias contribute dierently for answer prediction, in this paper, we overcome the language prior problem by proposing a novel Language Bias driven Curriculum Learning (LBCL) approach, which employs an easy-to-hard learning strategy with a novel diculty metric Visual Sensitive Coecient (VSC). Specically, in the initial training stage, the VQA model mainly learns the supercial textual correlations between questions and answers (easy concept) from more-biased examples, and then progressively focuses on learning the multimodal reasoning (hard concept) from less-biased examples in the following stages. The curriculum selection of examples on dierent stages is according to our proposed diculty metric VSC, which is to evaluate the diculty driven by the language bias of each VQA sample. Furthermore, to avoid the catastrophic forgetting of the learned concept during the multi-stage learning procedure, we propose to integrate knowledge distillation into the curriculum learning framework. Extensive experiments show that our LBCL can be generally applied to common VQA baseline models, and achieves remarkably better performance on the VQA-CP v1 and v2 datasets, with an overall 20% accuracy boost over baseline models. CCS CONCEPTS• Computing methodologies ! Computer vision tasks.

show abstract

“…Existing works to address this can be categorized into three groups. The first group attempts to reduce the language bias by designing new VQA models or learning strategies Ramakrishnan et al, 2018;Cadene et al, 2019;Clark et al, 2019;Grand and Belinkov, 2019;Clark et al, 2019;Jing et al, 2020;Niu et al, 2021;Gat et al, 2020). For example, RUBi and Ensemble (Cadene et al, 2019;Clark et al, 2019) explicitly modeled the questionanswer correlations to encourage VQA models to explore other patterns in the data that are more likely to generalize.…”

Section: Related Workmentioning

confidence: 99%

Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answering

Kil¹,

Zhang²,

Xuan³

et al. 2021

Preprint

View full text Add to dashboard Cite

Visual question answering (VQA) is challenging not only because the model has to handle multi-modal information, but also because it is just so hard to collect sufficient training examples -there are too many questions one can ask about an image. As a result, a VQA model trained solely on human-annotated examples could easily over-fit specific question styles or image contents that are being asked, leaving the model largely ignorant about the sheer diversity of questions. Existing methods address this issue primarily by introducing an auxiliary task such as visual grounding, cycle consistency, or debiasing. In this paper, we take a drastically different approach. We found that many of the "unknowns" to the learned VQA model are indeed "known" in the dataset implicitly. For instance, questions asking about the same object in different images are likely paraphrases; the number of detected or annotated objects in an image already provides the answer to the "how many" question, even if the question has not been annotated for that image. Building upon these insights, we present a simple data augmentation pipeline SIMPLEAUG to turn this "known" knowledge into training examples for VQA. We show that these augmented examples can notably improve the learned VQA models' performance, not only on the VQA-CP dataset with language prior shifts but also on the VQA v2 dataset without such shifts. Our method further opens up the door to leverage weakly-labeled or unlabeled images in a principled way to enhance VQA models. Our code and data are publicly available at https://github.com/ heendung/simpleAUG.

show abstract

Overcoming Language Priors in VQA via Decomposed Linguistic Representations

Cited by 73 publications

References 24 publications

Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods

Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods

From Superficial to Deep: Language Bias driven Curriculum Learning for Visual Question Answering

Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answering

Contact Info

Product

Resources

About