Causal Transportability for Visual Recognition

Mao, Chengzhi; Xia, Kevin; Wang, James; Wang, Hao; Yang, Junfeng; Bareinboim, Elias; Vondrick, Carl

doi:10.1109/cvpr52688.2022.00737

Cited by 21 publications

(7 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…image backdrop or contextual scene) that an image is placed in. This is a pernicious cause of disparity in algorithm performance and is widely investigated as an out-of-distribution generalisation problem in computer vision [24]. For the specific computer vision problem of facial affect recognition, the biggest dichotomy is between that of a "lab-controlled" and an "in-the-wild" dataset.…”

Section: Contextual Biasmentioning

confidence: 99%

Causal Structure Learning of Bias for Fair Affect Recognition

Cheong

Kalkan

Güneş

2023

2023 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW)

View full text Add to dashboard Cite

The problem of bias in facial affect recognition tools can lead to severe consequences and issues. It has been posited that causality is able to address the gaps induced by the associational nature of traditional machine learning, and one such gap is that of fairness. However, given the nascency of the field, there is still no clear mapping between tools in causality and applications in fair machine learning for the specific task of affect recognition. To address this gap, we provide the first causal structure formalisation of the different biases that can arise in affect recognition. We conducted a proof of concept on utilising causal structure learning for the post-hoc understanding and analysing bias.

show abstract

Section: Contextual Biasmentioning

confidence: 99%

Causal Structure Learning of Bias for Fair Affect Recognition

Cheong

Kalkan

Güneş

2023

2023 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW)

View full text Add to dashboard Cite

show abstract

“…The spurious label of each training example (e.g., whether this example contains the spurious feature) is either provided (Sagawa et al, 2019;Izmailov et al, 2022) or inferred by training a reference model (Nam et al, 2020;Creager et al, 2021;Liu et al, 2021;Nam et al, 2022) until it learns the spurious correlations. Other approaches indirectly estimate and use the causal effect of hidden non-labeled spurious attributes in pre-training (Mao et al, 2022).…”

Section: Related Workmentioning

confidence: 99%

“…In our experiments, we used GradCAM (Selvaraju et al, 2017) for the explanation maps. While GradCAM explanations may not be perfectly aligned with the model's attention, their usage has shown practical benefits for model debugging (Yosinski et al, 2016;Simonyan et al, 2014;Mao et al, 2022).…”

Section: Metricsmentioning

confidence: 99%

“…Another dataset we considered for evaluation is CelebA (Liu et al, 2015) for the task of hair color classification. Previous work (Mao et al, 2022;Sagawa et al, 2019) has shown that models trained on such data can have a lower accuracy for small groups defined by the gender attribute such as men with blond hair, since this group has a low representation in the training data. It turns out however that model accuracy does not degrade for this group using CLIP model, which is why we do not present results on CelebA in this paper.…”

Section: Datasetsmentioning

confidence: 99%

See 1 more Smart Citation

Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models

Wang

Yang

Wang

2022

Findings of the Association for Computational Linguistics: NAACL 2022

View full text Add to dashboard Cite

Recently, NLP models have achieved remarkable progress across a variety of tasks; however, they have also been criticized for being not robust. Many robustness problems can be attributed to models exploiting spurious correlations, or shortcuts between the training data and the task labels. Most existing work identifies a limited set of task-specific shortcuts via human priors or error analyses, which requires extensive expertise and efforts. In this paper, we aim to automatically identify such spurious correlations in NLP models at scale. We first leverage existing interpretability methods to extract tokens that significantly affect model's decision process from the input text. We then distinguish "genuine" tokens and "spurious" tokens by analyzing model predictions across multiple corpora and further verify them through knowledge-aware perturbations. We show that our proposed method can effectively and efficiently identify a scalable set of "shortcuts", and mitigating these leads to more robust models in multiple applications.

show abstract

“…(Zhang et al 2022a) assigned high confidence to image regions consistent with the global semantics in aggregating. (Pan, Wu, and Zhang 2023) proposed to eliminate redundant or irrelevant fragment alignments from the perspective of information coding. In general, not all fragments contribute to image-text relevance, and a large branch of existing methods is devoted to mining the vital ones to measure the relevance accurately.…”

Section: Introductionmentioning

confidence: 99%

Identification of Necessary Semantic Undertakers in the Causal View for Image-Text Matching

Zhang,

Zhang

et al. 2024

AAAI

View full text Add to dashboard Cite

Image-text matching bridges vision and language, which is a fundamental task in multimodal intelligence. Its key challenge lies in how to capture visual-semantic relevance. Fine-grained semantic interactions come from fragment alignments between image regions and text words. However, not all fragments contribute to image-text relevance, and many existing methods are devoted to mining the vital ones to measure the relevance accurately. How well image and text relate depends on the degree of semantic sharing between them. Treating the degree as an effect and fragments as its possible causes, we define those indispensable causes for the generation of the degree as necessary undertakers, i.e., if any of them did not occur, the relevance would be no longer valid. In this paper, we revisit image-text matching in the causal view and uncover inherent causal properties of relevance generation. Then we propose a novel theoretical prototype for estimating the probability-of-necessity of fragments, PN_f, for the degree of semantic sharing by means of causal inference, and further design a Necessary Undertaker Identification Framework (NUIF) for image-text matching, which explicitly formalizes the fragment's contribution to image-text relevance by modeling PN_f in two ways. Extensive experiments show our method achieves state-of-the-art on benchmarks Flickr30K and MSCOCO.

show abstract

Causal Transportability for Visual Recognition

Cited by 21 publications

References 20 publications

Causal Structure Learning of Bias for Fair Affect Recognition

Causal Structure Learning of Bias for Fair Affect Recognition

Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models

Identification of Necessary Semantic Undertakers in the Causal View for Image-Text Matching

Contact Info

Product

Resources

About