OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization

Ye, Nanyang; Li, Kaican; Bai, Haili; Runpeng, Yu,; Hong, Liang; Zhou, Fengwei; Li, Zhenguo; Zhu, Jun

doi:10.1109/cvpr52688.2022.00779

Cited by 48 publications

(57 citation statements)

References 74 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The inner mechanism of the labeling rule G → Y usually depends on the causal features, which are particular subparts of the entire data (Arjovsky et al, 2019;Kaddour et al, 2022;Wu et al, 2022b;Ye et al, 2022;Lu et al, 2021). While their complementary parts, environmental features, are noncausal for predicting the graphs.…”

Section: Definitions and Problem Formationsmentioning

confidence: 99%

“…Hence, OOD generalization on graphs is attracting widespread attention (Li et al, 2022b). However, existing studies mostly focus on correlation shift, which is just one type of OOD issue (Ye et al, 2022;Wiles et al, 2022). While another type, covariate shift, remains largely unexplored but is the focus of our work.…”

mentioning

confidence: 99%

“…Following prior studies (Arjovsky et al, 2019;Wu et al, 2022b), we assume causal features are stable and invariant across distributions, in contrast to the environmental features. Correlation shift denotes that environments and labels establish inconsistent statistical correlations in training and test data; whereas, covariate shift means that the environmental features in test data are unseen in training data (Ye et al, 2022;Wiles et al, 2022;Gui et al, 2022). For example, in Figure 1, the environmental features ladder and tree are different in training and test data, which forms the covariate shift (↔).…”

mentioning

confidence: 99%

See 2 more Smart Citations

Adversarial Causal Augmentation for Graph Covariate Shift

Sui¹,

Wang²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Out-of-distribution (OOD) generalization on graphs is drawing widespread attention. However, existing efforts mainly focus on the OOD issue of correlation shift. While another type, covariate shift, remains largely unexplored but is the focus of this work. From a data generation view, causal features are stable substructures in data, which play key roles in OOD generalization. While their complementary parts, environments, are unstable features that often lead to various distribution shifts. Correlation shift establishes spurious statistical correlations between environments and labels. In contrast, covariate shift means that there exist unseen environmental features in test data. Existing strategies of graph invariant learning and data augmentation suffer from limited environments or unstable causal features, which greatly limits their generalization ability on covariate shift. In view of that, we propose a novel graph augmentation strategy: Adversarial Causal Augmentation (AdvCA), to alleviate the covariate shift. Specifically, it adversarially augments the data to explore diverse distributions of the environments. Meanwhile, it keeps the causal features invariant across diverse environments. It maintains the environmental diversity while ensuring the invariance of the causal features, thereby effectively alleviating the covariate shift. Extensive experimental results with in-depth analyses demonstrate that AdvCA can outperform 14 baselines on synthetic and real-world datasets with various covariate shifts.

show abstract

Section: Definitions and Problem Formationsmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Adversarial Causal Augmentation for Graph Covariate Shift

Sui¹,

Wang²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…[30] compiles ten benchmarks of realistic distribution shifts over diverse applications (medical, economic, etc). Many algorithms have been proposed to improve out-of-distribution robustness [38,49,2,33], though in comprehensive evaluations, their gains over empirical risk minimization are marginal, as they often only hold for certain distribution shifts [70,21]. [70] identifies diversity and correlation shifts as two key dimensions to OOD robustness; our work focuses on the latter.…”

Section: Review Of Literaturementioning

confidence: 99%

Explicit Tradeoffs between Adversarial and Natural Distributional Robustness

Moayeri¹,

Banihashem²,

Feizi³

2022

Preprint

View full text Add to dashboard Cite

Several existing works study either adversarial or natural distributional robustness of deep neural networks separately. In practice, however, models need to enjoy both types of robustness to ensure reliability. In this work, we bridge this gap and show that in fact, explicit tradeoffs exist between adversarial and natural distributional robustness. We first consider a simple linear regression setting on Gaussian data with disjoint sets of core and spurious features. In this setting, through theoretical and empirical analysis, we show that (i) adversarial training with 1 and 2 norms increases the model reliance on spurious features; (ii) For ∞ adversarial training, spurious reliance only occurs when the scale of the spurious features is larger than that of the core features; (iii) adversarial training can have an unintended consequence in reducing distributional robustness, specifically when spurious correlations are changed in the new test domain. Next, we present extensive empirical evidence, using a test suite of twenty adversarially trained models evaluated on five benchmark datasets (ObjectNet, RIVAL10, Salient ImageNet-1M, ImageNet-9, Waterbirds), that adversarially trained classifiers rely on backgrounds more than their standardly trained counterparts, validating our theoretical results. We also show that spurious correlations in training data (when preserved in the test domain) can improve adversarial robustness, revealing that previous claims that adversarial vulnerability is rooted in spurious correlations are incomplete.

show abstract

“…Although data Independent and Identically Distributed (IID) is a primary assumption behind most machine learning systems, it does not hold in many real-world scenarios due to continuous distribution shifts [39,88]. Machine learning models encounter serious performance degradation [9,32,34] in such Out-of-Distribution (OoD) scenarios.…”

Section: Introductionmentioning

confidence: 99%

ZooD: Exploiting Model Zoo for Out-of-Distribution Generalization

Dong¹,

Awais²,

Zhou³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent advances on large-scale pre-training have shown great potentials of leveraging a large set of Pre-Trained Models (PTMs) for improving Out-of-Distribution (OoD) generalization, for which the goal is to perform well on possible unseen domains after fine-tuning on multiple training domains. However, maximally exploiting a zoo of PTMs is challenging since fine-tuning all possible combinations of PTMs is computationally prohibitive while accurate selection of PTMs requires tackling the possible data distribution shift for OoD tasks. In this work, we propose ZooD, a paradigm for PTMs ranking and ensemble with feature selection. Our proposed metric ranks PTMs by quantifying inter-class discriminability and inter-domain stability of the features extracted by the PTMs in a leave-one-domain-out cross-validation manner. The top-K ranked models are then aggregated for the target OoD task. To avoid accumulating noise induced by model ensemble, we propose an efficient variational EM algorithm to select informative features. We evaluate our paradigm on a diverse model zoo consisting of 35 models for various OoD tasks and demonstrate: (i) model ranking is better correlated with fine-tuning ranking than previous methods and up to 9859x faster than brute-force fine-tuning; (ii) OoD generalization after model ensemble with feature selection outperforms the state-of-the-art methods and the accuracy on most challenging task DomainNet is improved from 46.5% to 50.6%. Furthermore, we provide the fine-tuning results of 35 PTMs on 7 OoD datasets, hoping to help the research of model zoo and OoD generalization. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/zood.

show abstract

OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization

Abstract: If it is the author's pre-published version, changes introduced as a result of publishing processes such as copy-editing and formatting may not be reflected in this document. For a definitive version of this work, please refer to the published version.

Cited by 48 publications

References 74 publications

Adversarial Causal Augmentation for Graph Covariate Shift

Adversarial Causal Augmentation for Graph Covariate Shift

Explicit Tradeoffs between Adversarial and Natural Distributional Robustness

ZooD: Exploiting Model Zoo for Out-of-Distribution Generalization

Contact Info

Product

Resources

About