On Guiding Visual Attention with Language Specification

Petryk, Suzanne; Dunlap, Lisa; Nasseri, Keyan; Darrell, Trevor; Rohrbach, Anna

doi:10.1109/cvpr52688.2022.01756

Cited by 10 publications

(2 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There is a line of recent work aiming to fix vision classifiers with language inputs. (Petryk et al, 2022) uses attention maps from a pre-trained CLIP to supervise a CNN classifier's spatial attention. (Zhang et al, 2023) probes a vision classifier trained on the joint vision-language embedding space of CLIP using language embeddings of attributes, identifies the attributes causing most failures, and generates a large set of natural language inputs with the influential attributes to rectify the model.…”

Section: Related Workmentioning

confidence: 99%

“…While the multi-modal alignment increases the expectations about model reliability due to better grounding and larger availability of data in general, these models are still not immune to fundamental learning problems such as dealing with spurious correlations (Bommasani et al, 2021;Moayeri et al, 2022;Petryk et al, 2022;Agarwal et al, 2021). Therefore, when such models are used as a backbone to solve application-oriented tasks on a given domain, existing spurious correlations specific to that domain or the finetuning data that comes with it, may resurface in ways that are harmful to end users.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models

Wang

Yang

Wang

2022

Findings of the Association for Computational Linguistics: NAACL 2022

View full text Add to dashboard Cite

Recently, NLP models have achieved remarkable progress across a variety of tasks; however, they have also been criticized for being not robust. Many robustness problems can be attributed to models exploiting spurious correlations, or shortcuts between the training data and the task labels. Most existing work identifies a limited set of task-specific shortcuts via human priors or error analyses, which requires extensive expertise and efforts. In this paper, we aim to automatically identify such spurious correlations in NLP models at scale. We first leverage existing interpretability methods to extract tokens that significantly affect model's decision process from the input text. We then distinguish "genuine" tokens and "spurious" tokens by analyzing model predictions across multiple corpora and further verify them through knowledge-aware perturbations. We show that our proposed method can effectively and efficiently identify a scalable set of "shortcuts", and mitigating these leads to more robust models in multiple applications.

show abstract

Section: Related Workmentioning

confidence: 99%