Domain-robust VQA with diverse datasets and methods but no target labels

Zhang, Mingda; Maidment, Tristan; Diab, Ahmed A. Zaki; Kovashka, Adriana; Hwa, Rebecca

doi:10.1109/cvpr46437.2021.00697

Cited by 15 publications

(7 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, when the exact nature of distribution shift between train and test splits is known (such as in (Agrawal et al, 2018)), approaches developed to tackle such shifts tend to rely on the explicit knowledge of construction of such OOD splits resulting in inflated sense of progress (Teney et al, 2020). Similar to us, Zhang et al (2021); Hudson and Manning (2019) also present some experimental results on VQA OOD evaluation, however they do it in limited manner (e.g., do not consider all pairs of datasets, do not evaluate the effect of multimodal pretraining, etc.). To our best knowledge, ours is the first work to extensively quantifying the extent of IID to OOD performance drops in current VQA models and study the effect of several factors: answer overlap, multimodal pretraining, generative vs. discriminative modeling, and stringent evaluation metric.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization

Agrawal,

Kajic,

Bugliarello

et al. 2023

Findings of the Association for Computational Linguistics: EACL 2023

View full text Add to dashboard Cite

Vision-and-language (V&L) models pretrained on large-scale multimodal data have demonstrated strong performance on various tasks such as image captioning and visual question answering (VQA). The quality of such models is commonly assessed by measuring their performance on unseen data that typically comes from the same distribution as the training data. However, when evaluated under out-of-distribution (out-of-dataset) settings for VQA, we observe that these models exhibit poor generalization. We comprehensively evaluate two pretrained V&L models under different settings (i.e. classification and open-ended text generation) by conducting cross-dataset evaluations. We find that these models tend to learn to solve the benchmark, rather than learning the high-level skills required by the VQA task. We also find that in most cases generative models are less susceptible to shifts in data distribution compared to discriminative ones, and that multimodal pretraining is generally helpful for OOD generalization. Finally, we revisit assumptions underlying the use of automatic VQA evaluation metrics, and empirically show that their stringent nature repeatedly penalizes models for correct responses. * denotes equal first author contribution. † denotes equal contribution. ‡ denotes equal senior contribution. Detailed contributions follow at the end of the paper.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Domain adaptation in VQA. Some studies (Jabri et al, 2016;Chao et al, 2018;Zhang et al, 2021) have explored domain adaptation of VQA models from one VQA benchmark to another. Our focus, instead, is on evaluating zero-shot crossbenchmark generalization without any adaptation.…”

Section: Related Workmentioning

confidence: 99%

Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization

Agrawal,

Kajic,

Bugliarello

et al. 2023

Findings of the Association for Computational Linguistics: EACL 2023

View full text Add to dashboard Cite

show abstract

“…However these VQA datasets build a closed world that is not designed to generalize to real world images. Remarkably, some recent work has managed to show domain transfer from cartoon images to real images [67], but there is still a limitation on how much could be learned from these existing resources. Our proposed Hypersim-VQA and ThreeDWorld-VQA datasets provide a promising alternative that more realistically captures real world settings and offers a path forward in this direction.…”

Section: Introductionmentioning

confidence: 99%

“…[67] Two-stage DA; in both cases the first number in E refers to the training epoch parameter for the AE. For Domain Independent, di tokens is the additional output we use for the synthetic answer tokens.Adversarial MMD lr = 15e − 4 E = 100 + 13 lr = 1e − 3 E = 150 + 13 O wd = 1e − 6 O = Adam O wd = 1e − 4 O = Adam O = 1e − 4 O β = (0.8, 0.8) O = 1e − 4 O β = (0.8, 0.8) α = 2 (1+exp(−10 * p))−1 α = 0.4 β = 0.6 Domain Independent F-SWAP lr = 15e − 4 E = 13 lr = 15e − 4 E = 13 O wd = 0.2 O = Adam O wd = 1e − 1 O = Adam O = 1e − 9 O β = (0.9, 0.9) O = 1e − 9 O β = (0.9, 0.98) di tokens = 100 β = 1. λ = 0.2…”

mentioning

confidence: 99%

SimVQA: Exploring Simulated Environments for Visual Question Answering

Cascante-Bonilla¹,

Wang²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Existing work on VQA explores data augmentation to achieve better generalization by perturbing images in the dataset or modifying existing questions and answers. While these methods exhibit good performance, the diversity of the questions and answers are constrained by the available images. In this work we explore using synthetic computergenerated data to fully control the visual and language space, allowing us to provide more diverse scenarios. We quantify the effectiveness of leveraging synthetic data for real-world VQA. By exploiting 3D and physics simulation platforms, we provide a pipeline to generate synthetic data to expand and replace type-specific questions and answers without risking exposure of sensitive or personal data that might be present in real images. We offer a comprehensive analysis while expanding existing hyper-realistic datasets to be used for VQA. We also propose Feature Swapping (F-SWAP) -where we randomly switch object-level features during training to make a VQA model more domain invariant. We show that F-SWAP is effective for improving VQA models on real images without compromising on their accuracy to answer existing questions in the dataset.

show abstract

“…With the introduction of image search by image, according to internet search giant, the application of IBL attracted widespread attention. Besides, these academic fields own high enthusiasm for researching IBL: object detection [1-3], visual localization [4][5][6], simultaneous localization and mapping (SLAM) [7], etc.…”

Section: Introductionmentioning

confidence: 99%

Street-Level Image Localization Based on Building-Aware Features via Patch-Region Retrieval under Metropolitan-Scale

et al. 2021

View full text Add to dashboard Cite

The aim of image-based localization (IBL) is to localize the real location of query image by matching reference image in database with GNSS-tags. Popular methods related to IBL commonly use street-level images, which have high value in practical application. Using street-level image to tackle IBL task has the primary challenges: existing works have not made targeted optimization for urban IBL tasks. Besides, the matching result is over-reliant on the quality of image features. Methods should address their practicality and robustness in engineering application, under metropolitan-scale. In response to these, this paper made following contributions: firstly, given the critical of buildings in distinguishing urban scenes, we contribute a feature called Building-Aware Feature (BAF). Secondly, in view of negative influence of complex urban scenes in retrieval process, we propose a retrieval method called Patch-Region Retrieval (PRR). To prove the effectiveness of BAF and PRR, we established an image-based localization experimental framework. Experiments prove that BAF can retain the feature points that fall on the building, and selectively lessen the feature points that fall on other things. While this effectively compresses the storage amount of feature index, we can also improve recall of localization results; implemented in the stage of geometric verification, PRR compares matching results of regional features and selects the best ranking as final result. PRR can enhance effectiveness of patch-regional feature. In addition, we fully confirmed the superiority of our proposed methods through a metropolitan-scale street-level image dataset.

show abstract

Domain-robust VQA with diverse datasets and methods but no target labels

Cited by 15 publications

References 40 publications

Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization

Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization

SimVQA: Exploring Simulated Environments for Visual Question Answering

Street-Level Image Localization Based on Building-Aware Features via Patch-Region Retrieval under Metropolitan-Scale

Contact Info

Product

Resources

About