An Online Learning Approach to Interpolation and Extrapolation in Domain Generalization

Rosenfeld, Elan; Ravikumar, Pradeep; Risteski, Andrej

doi:10.48550/arxiv.2102.13128

Cited by 5 publications

(7 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent works, [Rosenfeld et al, 2021b, Gulrajani and Lopez-Paz, 2021, Rosenfeld et al, 2021a, Kamath et al, 2021, different limitations of invariance based approaches in addressing OOD generalization failures were highlighted. In [Rosenfeld et al, 2021b], the authors showed that if we use the IRMv1 objective, then for non-linear tasks the solutions from IRMv1 are no better than ERM in generalizing OOD.…”

Section: Invariance Principles In Ood Generalizationmentioning

confidence: 99%

Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization

Ahuja¹,

Caballero²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

The invariance principle from causality is at the heart of notable approaches such as invariant risk minimization (IRM) that seek to address out-of-distribution (OOD) generalization failures. Despite the promising theory, invariance principle-based approaches fail in common classification tasks, where invariant (causal) features capture all the information about the label. Are these failures due to the methods failing to capture the invariance? Or is the invariance principle itself insufficient? To answer these questions, we revisit the fundamental assumptions in linear regression tasks, where invariance-based approaches were shown to provably generalize OOD. In contrast to the linear regression tasks, we show that for linear classification tasks we need much stronger restrictions on the distribution shifts, or otherwise OOD generalization is impossible. Furthermore, even with appropriate restrictions on distribution shifts in place, we show that the invariance principle alone is insufficient. We prove that a form of the information bottleneck constraint along with invariance helps address key failures when invariant features capture all the information about the label and also retains the existing success when they do not. We propose an approach that incorporates both of these principles and demonstrate its effectiveness in several experiments.

show abstract

Section: Invariance Principles In Ood Generalizationmentioning

confidence: 99%

Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization

Ahuja¹,

Caballero²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…[2019] give a generalization bound for distributions with sufficiently small H-divergence, while Rosenfeld et al [2021a] model domain generalization as an online game, showing that generalizing beyond the convex hull is NP-hard.…”

Section: Additional Related Workmentioning

confidence: 99%

Iterative Feature Matching: Toward Provable Domain Generalization with Logarithmic Environments

Rosenfeld¹,

Sellke²,

Ma³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Domain generalization aims at performing well on unseen test environments with data from a limited number of training environments. Despite a proliferation of proposal algorithms for this task, assessing their performance, both theoretically and empirically is still very challenging. Moreover, recent approaches such as Invariant Risk Minimization (IRM) require a prohibitively large number of training environments -linear in the dimension of the spurious feature space ds-even on simple data models like the one proposed by Rosenfeld et al. [2021b]. Under a variant of this model, we show that both ERM and IRM cannot generalize with o(ds) environments. We then present a new algorithm based on performing iterative feature matching that is guaranteed with high probability to yield a predictor that generalizes after seeing only O(log ds) environments.

show abstract

“…The DG problem setting was first analysed in (Blanchard et al, 2011). Since then there have been some attempts to analyse DG algorithms from a generalisation bound perspective (Muandet et al, 2013;Blanchard et al, 2021;Hu et al, 2020;Albuquerque et al, 2020;Rosenfeld et al, 2021). However these studies have theoretical results that are either restricted to specific model classes, such as kernel machines, or make strong assumptions about how the domains seen during training will resemble those seen at test time-e.g., that all domains are convex combinations of a finite predetermined set of prototypical domains.…”

Section: Related Work Theoretical Analysis Of the Dg Setting And Algo...mentioning

confidence: 99%

“…However, we also provide a means to transform any bound on the expected (or "average-case") risk to a high-confidence bound on the worst-case risk. Rosenfeld et al (2021) is another piece of work that theoretically investigates the generalisation of ERM in a DG setting. They deal with online DG, where each time-step corresponds to observing a new domain, and the learner must produce a new model capable of generalising to novel domains.…”

Section: Related Work Theoretical Analysis Of the Dg Setting And Algo...mentioning

confidence: 99%

“…This object is also the most commonly analysed idealised objective in the learning theory literature (Blanchard et al, 2011;Muandet et al, 2013;Blanchard et al, 2021;Rosenfeld et al, 2021), but other formulations also exist (Arjovsky et al, 2019). As with the single domain learning problem, we only have access to an empirical estimate of the risk,…”

Section: Bounding Risk For Domain Generalizationmentioning

confidence: 99%

See 1 more Smart Citation

Finding lost DG: Explaining domain generalization via model complexity

Li¹,

Gouk²,

Hospedales³

2022

Preprint

View full text Add to dashboard Cite

The domain generalization (DG) problem setting challenges a model trained on multiple known data distributions to generalise well on unseen data distributions. Due to its practical importance, a large number of methods have been proposed to address this challenge. However much of the work in general purpose DG is heuristically motivated, as the DG problem is hard to model formally; and recent evaluations have cast doubt on existing methods' practical efficacy -in particular compared to a well tuned empirical risk minimisation baseline. We present a novel learning-theoretic generalisation bound for DG that bounds unseen domain generalisation performance in terms of the model's Rademacher complexity. Based on this, we conjecture that existing methods' efficacy or lack thereof is largely determined by an empirical risk vs predictor complexity trade-off, and demonstrate that their performance variability can be explained in these terms. Algorithmically, this analysis suggests that domain generalisation should be achieved by simply performing regularised ERM with a leave-one-domain-out crossvalidation objective. Empirical results on the Do-mainBed benchmark corroborate this.

show abstract

An Online Learning Approach to Interpolation and Extrapolation in Domain Generalization

Cited by 5 publications

References 17 publications

Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization

Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization

Iterative Feature Matching: Toward Provable Domain Generalization with Logarithmic Environments

Finding lost DG: Explaining domain generalization via model complexity

Contact Info

Product

Resources

About