Check It Again:Progressive Visual Question Answering via Visual Entailment

Si, Qingyi; Lin, Zheng; M, Zheng; Fu, Peng; Wang, Weiping

doi:10.18653/v1/2021.acl-long.317

Cited by 24 publications

(7 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, some studies [23] have found that this approach removes the good language prior required by the model. Alternatively, SAR [24] uses an additional model for answer filtering by adding semantic information about the answers to the model. However, we found that this method makes the model computationally intensive and uses a pre-trained model LXMERT [11] that uses a partial sample of the VQA-CP v2 test set.…”

Section: Methods Modifying Model Architecturementioning

confidence: 99%

“…We implemented our approach on the baseline models of UpDn [10], CSS [27], LXMERT [34], and VLMO [12], respectively, and compared it with other mainstream approaches, namely SAR [24], Rescaling [32], CL [38], CSS þ IntorD [22], DM [28], LMH-AttReg [26], and VPCL [25]. The VLMO model employs a shared embedding space to process images and text.…”

Section: Baseline Modelsmentioning

confidence: 99%

See 1 more Smart Citation

StableNet: Distinguishing the hard samples to overcome language priors in visual question answering

Yu,

Zhao,

Guo

et al. 2023

IET Computer Vision

View full text Add to dashboard Cite

With the booming fields of computer vision and natural language processing, cross‐modal intersections such as visual question answering (VQA) have become very popular. However, several studies have shown that many VQA models suffer from severe language prior problems. After a series of experiments, the authors found that previous VQA models are in an unstable state, that is, when training is repeated several times on the same dataset, there are significant differences between the distributions of the predicted answers given by the models each time, and these models also perform unsatisfactorily in terms of accuracy. The reason for model instability is that some of the difficult samples bring serious interference to model training, so we design a method to measure model stability quantitatively and further propose a method that can alleviate both model imbalance and instability phenomena. Precisely, the question types are classified into simple and difficult ones different weighting measures are applied. By imposing constraints on the training process for both types of questions, the stability and accuracy of the model improve. Experimental results demonstrate the effectiveness of our method, which achieves 63.11% on VQA‐CP v2 and 75.49% with the addition of the pre‐trained model.

show abstract

Section: Methods Modifying Model Architecturementioning

confidence: 99%

Section: Baseline Modelsmentioning

confidence: 99%

StableNet: Distinguishing the hard samples to overcome language priors in visual question answering

Yu,

Zhao,

Guo

et al. 2023

IET Computer Vision

View full text Add to dashboard Cite

show abstract

“…Third, adversarial methods [ 19 ], such as using adversarial losses, are used to reduce known sources of bias by inducing errors in the model when it is presented with only the question. Fourth, contrast learning-based methods [ 4 , 20 , 21 , 22 ] are used to enhance the utilization of information between the visual context and the question by constructing negative image-question pairs. Fifth, an additional annotation-based approach, refs.…”

Section: Related Workmentioning

confidence: 99%

Collaborative Modality Fusion for Mitigating Language Bias in Visual Question Answering

Lu,

Chen,

Zhu

2024

J. Imaging

View full text Add to dashboard Cite

Language bias stands as a noteworthy concern in visual question answering (VQA), wherein models tend to rely on spurious correlations between questions and answers for prediction. This prevents the models from effectively generalizing, leading to a decrease in performance. In order to address this bias, we propose a novel modality fusion collaborative de-biasing algorithm (CoD). In our approach, bias is considered as the model’s neglect of information from a particular modality during prediction. We employ a collaborative training approach to facilitate mutual modeling between different modalities, achieving efficient feature fusion and enabling the model to fully leverage multimodal knowledge for prediction. Our experiments on various datasets, including VQA-CP v2, VQA v2, and VQA-VS, using different validation strategies, demonstrate the effectiveness of our approach. Notably, employing a basic baseline model resulted in an accuracy of 60.14% on VQA-CP v2.

show abstract

“…Therefore, we do not directly compare experimentally with this line of works. Most recently, another line of work has been released where an additional objective of Visual Entailment is added to further boost performance (Si et al, 2021), which we do not compare for fairness.…”

Section: Related Workmentioning

confidence: 99%

Generative Bias for Visual Question Answering

Cho¹,

Kim²,

Ryu³

et al. 2022

Preprint

View full text Add to dashboard Cite

The task of Visual Question Answering (VQA) is known to be plagued by the issue of VQA models exploiting biases within the dataset to make its final prediction. Many previous ensemble based debiasing methods have been proposed where an additional model is purposefully trained to be biased in order to aid in training a robust target model. However, these methods compute the bias for a model from the label statistics of the training data or directly from single modal branches. In contrast, in this work, in order to better learn the bias a target VQA model suffers from, we propose a generative method to train the bias model directly from the target model, called GenB. In particular, GenB employs a generative network to learn the bias through a combination of the adversarial objective and knowledge distillation. We then debias our target model with GenB as a bias model, and show through extensive experiments the effects of our method on various VQA bias datasets including VQA-CP2, VQA-CP1, GQA-OOD, and VQA-CE.

show abstract

Check It Again:Progressive Visual Question Answering via Visual Entailment

Cited by 24 publications

References 24 publications

StableNet: Distinguishing the hard samples to overcome language priors in visual question answering

StableNet: Distinguishing the hard samples to overcome language priors in visual question answering

Collaborative Modality Fusion for Mitigating Language Bias in Visual Question Answering

Generative Bias for Visual Question Answering

Contact Info

Product

Resources

About