Gradient Matching for Domain Generalization

Shi, Yuge; Seely, Jeffrey; Torr, Philip H. S.; Siddharth, N.; Hannun, Awni; Usunier, Nicolas; Synnaeve, Gabriel

doi:10.48550/arxiv.2104.09937

Cited by 19 publications

(35 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By further modifying the learning rate from 1e-4 to 3e-4 and increasing the checkpoint frequency to be every 200 steps instead of 1000, we finally obtain a worst region accuracy of 34.8% (rows "+ higher lr"). We note that these final OOD test accuracies we obtain are competitive compared to the number reported 2 by Shi et al [9], 34.6%, achieved using a technique specifically designed for the domain shift problem (a method called Fish; last row in Table 2). Such "trivial" improvements over the baseline call for further tuning of the baseline settings before developing and evaluating new models on this dataset.…”

Section: Core Findingsmentioning

confidence: 45%

“…Nowadays, few researchers are surprised by NNs performing very well on some in-domain data distribution, and there has been increasing interest in developing models that are robust to domain shifts [8,9,10,11]. Here we focus on the recently proposed benchmark for evaluating domain robust systems, WILDS [11], and we share our empirical experience with two datasets of WILDS, iWildCam and FMoW, as well as their baseline models.…”

Section: Introductionmentioning

confidence: 99%

“…These numbers were taken from https://wilds.stanford.edu/leaderboard/ as recommended by Shi et al[9] (personal communication).…”

mentioning

confidence: 99%

See 2 more Smart Citations

Improving Baselines in the Wild

Irie¹,

Schlag²,

Csordás³

et al. 2021

Preprint

View full text Add to dashboard Cite

We share our experience with the recently released WILDS benchmark, a collection of ten datasets dedicated to developing models and training strategies which are robust to domain shifts. Several experiments yield a couple of critical observations which we believe are of general interest for any future work on WILDS. Our study focuses on two datasets: iWildCam and FMoW. We show that (1) Conducting separate cross-validation for each evaluation metric is crucial for both datasets, (2) A weak correlation between validation and test performance might make model development difficult for iWildCam, (3) Minor changes in the training of hyperparameters improve the baseline by a relatively large margin (mainly on FMoW), (4) There is a strong correlation between certain domains and certain target labels (mainly on iWildCam). To the best of our knowledge, no prior work on these datasets has reported these observations despite their obvious importance. Our code is public. 1

show abstract

Section: Core Findingsmentioning

confidence: 45%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Improving Baselines in the Wild

Irie¹,

Schlag²,

Csordás³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…( 4), one has to compute second order derivatives such as ∇ θ ∇ x log p Z|X (z|x), which is computationally prohibitive for tasks with large dimensional inputs such as stereo matching, semantic segmentation, etc. [38]. To overcome this issue, we propose ITSA, a simple yet computationally feasible approach to promote the learning of shortcut-invariant features.…”

Section: Robust Information Bottleneck and Fisher Informationmentioning

confidence: 99%

ITSA: An Information-Theoretic Approach to Automatic Shortcut Avoidance and Domain Generalization in Stereo Matching Networks

Chuah¹,

Tennakoon²,

Hoseinnezhad³

et al. 2022

Preprint

View full text Add to dashboard Cite

State-of-the-art stereo matching networks trained only on synthetic data often fail to generalize to more challenging real data domains. In this paper, we attempt to unfold an important factor that hinders the networks from generalizing across domains: through the lens of shortcut learning. We demonstrate that the learning of feature representations in stereo matching networks is heavily influenced by synthetic data artefacts (shortcut attributes). To mitigate this issue, we propose an Information-Theoretic Shortcut Avoidance (ITSA) approach to automatically restrict shortcutrelated information from being encoded into the feature representations. As a result, our proposed method learns robust and shortcut-invariant features by minimizing the sensitivity of latent features to input variations. To avoid the prohibitive computational cost of direct input sensitivity optimization, we propose an effective yet feasible algorithm to achieve robustness. We show that using this method, stateof-the-art stereo matching networks that are trained purely on synthetic data can effectively generalize to challenging and previously unseen real data scenarios. Importantly, the proposed method enhances the robustness of the synthetic trained networks to the point that they outperform their finetuned counterparts (on real data) for challenging out-ofdomain stereo datasets.

show abstract

“…Domain alignment methods attempt to learn a domain-invariant representation of the data from the source domains by regularizing the learning objective. Variants of such a regularization include the minimization across the source domains of the maximum mean discrepancy criteria (MMD) [21,35], the minimization of a distance metric between the domain-specific means [71] or covariance matrices [69], the minimization of a contrastive loss [50,83,44,29], or the maximization of loss gradient alignment [65,63]. Other works use adversarial training with a domain discriminator model [18,37] for the same purpose.…”

Section: Domain Generalizationmentioning

confidence: 99%

Towards Data-Free Domain Generalization

Frikha¹,

Chen²,

Krompaß³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this work, we investigate the unexplored intersection of domain generalization and data-free learning. In particular, we address the question: How can knowledge contained in models trained on different source data domains can be merged into a single model that generalizes well to unseen target domains, in the absence of source and target domain data? Machine learning models that can cope with domain shift are essential for for real-world scenarios with often changing data distributions. Prior domain generalization methods typically rely on using source domain data, making them unsuitable for private decentralized data. We define the novel problem of Data-Free Domain Generalization (DFDG), a practical setting where models trained on the source domains separately are available instead of the original datasets, and investigate how to effectively solve the domain generalization problem in that case. We propose DEKAN, an approach that extracts and fuses domain-specific knowledge from the available teacher models into a student model robust to domain shift. Our empirical evaluation demonstrates the effectiveness of our method which achieves first state-of-the-art results in DFDG by significantly outperforming ensemble and data-free knowledge distillation baselines.

show abstract

Gradient Matching for Domain Generalization

Cited by 19 publications

References 17 publications

Improving Baselines in the Wild

Improving Baselines in the Wild

ITSA: An Information-Theoretic Approach to Automatic Shortcut Avoidance and Domain Generalization in Stereo Matching Networks

Towards Data-Free Domain Generalization

Contact Info

Product

Resources

About