Learning Representations that Support Robust Transfer of Predictors

Xu, Yilun; Jaakkola, Tommi S.

doi:10.48550/arxiv.2110.09940

Cited by 4 publications

(4 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Arjovsky et al [2019] propose invariant risk minimization (IRM) to capture invariant correlations by learning representations that elicit an optimal invariant predictor across multiple training environments. Ahuja et al [2020], Jin et al [2020], Krueger et al [2021], Xie et al [2020] further develop several variants of IRM by introducing game theory, regret minimization, variance penalization, etc., and Xu and Jaakkola [2021], Chang et al [2020], Lin et al [2021] try to learn invariant features by coupled adversarial neural networks.…”

Section: Introductionmentioning

confidence: 99%

ZIN: When and How to Learn Invariance by Environment Inference?

Ye¹,

Zhu²,

Cui³

2022

Preprint

View full text Add to dashboard Cite

It is commonplace to encounter heterogeneous data, of which some aspects of the data distribution may vary but the underlying causal mechanisms remain constant. When data are divided into distinct environments according to the heterogeneity, recent invariant learning methods have proposed to learn robust and invariant models based on this environment partition. It is hence tempting to utilize the inherent heterogeneity even when environment partition is not provided. Unfortunately, in this work, we show that learning invariant features under this circumstance is fundamentally impossible without further inductive biases or additional information. Then, we propose a framework to jointly learn environment partition and invariant representation, assisted by additional auxiliary information. We derive sufficient and necessary conditions for our framework to provably identify invariant features under a fairly general setting. Experimental results on both synthetic and real world datasets validate our analysis and demonstrate an improved performance of the proposed framework over existing methods. Finally, our results also raise the need of making the role of inductive biases more explicit in future works, when considering learning invariant models without environment partition.

show abstract

Section: Introductionmentioning

confidence: 99%

ZIN: When and How to Learn Invariance by Environment Inference?

Ye¹,

Zhu²,

Cui³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For context biased datasets, we compare with Rebias [4], End [57], LfF [41], and Feat-Aug [27]. For domain gap dataset (DG task), we compare with domainlabel based methods, such as DANN [1], fish [53], and TRM [67], as well as domain-label free methods, such as RSC [21] and StableNet [70]. As we claimed at the end of Section 3.1, we train all models from scratch.…”

Section: Datasetmentioning

confidence: 99%

Class Is Invariant to Context and Vice Versa: On Learning Invariance for Out-Of-Distribution Generalization

Qi¹,

Tang²,

Sun³

et al. 2022

Preprint

View full text Add to dashboard Cite

Out-Of-Distribution generalization (OOD) is all about learning invariance against environmental changes. If the context 1 in every class is evenly distributed, OOD would be trivial because the context can be easily removed due to an underlying principle: class is invariant to context. However, collecting such a balanced dataset is impractical. Learning on imbalanced data makes the model bias to context and thus hurts OOD. Therefore, the key to OOD is context balance. We argue that the widely adopted assumption in prior work-the context bias can be directly annotated or estimated from biased class prediction-renders the context incomplete or even incorrect. In contrast, we point out the everoverlooked other side of the above principle: context is also invariant to class, which motivates us to consider the classes (which are already labeled) as the varying environments 2 to resolve context bias (without context labels). We implement this idea by minimizing the contrastive loss of intra-class sample similarity while assuring this similarity to be invariant across all classes. On benchmarks with various context biases and domain gaps, we show that a simple re-weighting based classifier equipped with our context estimation achieves state-of-the-art performance. We provide the theoretical justifications in Appendix and codes on Github 3 .

show abstract

“…The difficulty is finding the characteristic points of the new task. Xu et al [81] introduced a simple, robust estimation criterion-transfer risk-specifically geared towards optimizing transfer to new environments. The criterion amounts to finding a representation that minimizes the risk of applying any optimal predictor trained on one environment to another.…”

Section: Data Migrationmentioning

confidence: 99%

Out-of-Distribution (OOD) Detection Based on Deep Learning: A Review

Cui

Wang

2022

Electronics

View full text Add to dashboard Cite

Out-of-Distribution (OOD) detection separates ID (In-Distribution) data and OOD data from input data through a model. This problem has attracted increasing attention in the area of machine learning. OOD detection has achieved good intrusion detection, fraud detection, system health monitoring, sensor network event detection, and ecosystem interference detection. The method based on deep learning is the most studied in OOD detection. In this paper, related basic information on OOD detection based on deep learning is described, and we categorize methods according to the training data. OOD detection is divided into supervised, semisupervised, and unsupervised. Where supervised data are used, the methods are categorized according to technical means: model-based, distance-based, and density-based. Each classification is introduced with background, examples, and applications. In addition, we present the latest applications of OOD detection based on deep learning and the problems and expectations in this field.

show abstract

Learning Representations that Support Robust Transfer of Predictors

Cited by 4 publications

References 26 publications

ZIN: When and How to Learn Invariance by Environment Inference?

ZIN: When and How to Learn Invariance by Environment Inference?

Class Is Invariant to Context and Vice Versa: On Learning Invariance for Out-Of-Distribution Generalization

Out-of-Distribution (OOD) Detection Based on Deep Learning: A Review

Contact Info

Product

Resources

About