Optimal Representations for Covariate Shift

Ruan, Yangjun; Dubois, Yann; Maddison, Chris J.

doi:10.48550/arxiv.2201.00057

Cited by 3 publications

(4 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Spurious correlations and representation learning. Prior work has often associated poor robustness to spurious correlations with the quality of representations learned by the model [4,6,84] and suggested that the entire model needs to be carefully trained to avoid relying on spurious features [e.g. 85,43,58,107,92,76,56].…”

Section: Discussionmentioning

confidence: 99%

“…Other papers proposed methods based on meta-learning the weights for a weighted loss [80] and group-agnostic adaptive regularization [15,14]. Related methods have been developed in several areas of machine learning, such as ML Fairness [22,30,47,77,2,46], domain adaptation [23,24] and domain generalization [9,65,57,29,84] including works on Invariant Risk Minimization and causality [75,4,53,5].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations

Kirichenko¹,

Izmailov²,

Wilson³

2022

Preprint

View full text Add to dashboard Cite

Neural network classifiers can largely rely on simple spurious features, such as backgrounds, to make predictions. However, even in these cases, we show that they still often learn core features associated with the desired attributes of the data, contrary to recent findings. Inspired by this insight, we demonstrate that simple last layer retraining can match or outperform state-of-the-art approaches on spurious correlation benchmarks, but with profoundly lower complexity and computational expenses. Moreover, we show that last layer retraining on large ImageNet-trained models can also significantly reduce reliance on background and texture information, improving robustness to covariate shift, after only minutes of training on a single GPU.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations

Kirichenko¹,

Izmailov²,

Wilson³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In addition, both M min cpt and M max cpt have close connections with DG performance as shown in Theorem 4.2 and Theorem 4.3 in Section 4.2. The covariate shift is widely discussed in recent literature [Duchi et al, 2020;Ruan et al, 2021; yet none of them give the quantification with function discrepancy, which favors the analysis of DG performance and shows remarkable properties when H is large (such as the function space supported by current deep models). The concept shift can be considered as the discrepancy between the labeling rule f tr on the training data and the labeling rule f te on the test data.…”

Section: Metrics For Covariate Shift and Concept Shiftmentioning

confidence: 99%

NICO++: Towards Better Benchmarking for Domain Generalization

Zhang¹,

He²,

Xu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Despite the remarkable performance that modern deep neural networks have achieved on independent and identically distributed (I.I.D.) data, they can crash under distribution shifts. Most current evaluation methods for domain generalization (DG) adopt the leave-one-out strategy as a compromise on the limited number of domains. We propose a large-scale benchmark with extensive labeled domains named NICO ++ ‡ along with more rational evaluation methods for comprehensively evaluating DG algorithms. To evaluate DG datasets, we propose two metrics to quantify covariate shift and concept shift, respectively. Two novel generalization bounds from the perspective of data construction are proposed to prove that limited concept shift and significant covariate shift favor the evaluation capability for generalization. Through extensive experiments, NICO ++ shows its superior evaluation capability compared with current DG datasets and its contribution in alleviating unfairness caused by the leak of oracle knowledge in model selection.

show abstract

“…Domain adaptation is another closely related problem setting. Domain adaptation (DA) methods require access to labeled source and unlabeled target domains during training and aim to improve target performance via a combination of distribution matching [18,50,43], self-training [46,31], data augmentation [8,38], and other regularizers. DA methods are typically challenging to train and require retraining for every new target domain.…”

Section: Related Workmentioning

confidence: 99%

Understanding new tasks through the lens of training data via exponential tilting

Maity¹,

Yurochkin²,

Banerjee³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deploying machine learning models on new tasks is a major challenge despite the large size of the modern training datasets. However, it is conceivable that the training data can be reweighted to be more representative of the new (target) task. We consider the problem of reweighing the training samples to gain insights into the distribution of the target task. Specifically, we formulate a distribution shift model based on the exponential tilt assumption and learn train data importance weights minimizing the KL divergence between labeled train and unlabeled target datasets. The learned train data weights can then be used for downstream tasks such as target performance evaluation, fine-tuning, and model selection. We demonstrate the efficacy of our method on WATERBIRDS and BREEDS benchmarks. 1

show abstract

Optimal Representations for Covariate Shift

Cited by 3 publications

References 27 publications

Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations

Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations

NICO++: Towards Better Benchmarking for Domain Generalization

Understanding new tasks through the lens of training data via exponential tilting

Contact Info

Product

Resources

About