Robust Domain Adaptation: Representations, Weights and Inductive Bias

Bouvier, Victor; Véry, Philippe; Chastagnol, Clément; Tami, Myriam; Hudelot, Céline

doi:10.48550/arxiv.2006.13629

Cited by 3 publications

(7 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Outstanding progress have been towards learning more domain transferable representations by looking for domain invariance. The tensorial product between representations and prediction promotes conditional domain invariance [41], the use of weights [10,62,7,14] has dramatically improved the problem of label shift theoretically described in [64], hallucinating consistent target samples [38], penalizing high singular values of batch of representations [12] or by enforcing the favorable inductive bias of consistence through various data augmentation in the target domain [45]. Recent works address the problem of adaptation without source data [37,61].…”

Section: Discussionmentioning

confidence: 99%

Bridging Few-Shot Learning and Adaptation: New Challenges of Support-Query Shift

Bennequin¹,

Bouvier²,

Tami³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Few-Shot Learning (FSL) algorithms have made substantial progress in learning novel concepts with just a handful of labelled data. To classify query instances from novel classes encountered at test-time, they only require a support set composed of a few labelled samples. FSL benchmarks commonly assume that those queries come from the same distribution as instances in the support set. However, in a realistic setting, data distribution is plausibly subject to change, a situation referred to as Distribution Shift (DS). The present work addresses the new and challenging problem of Few-Shot Learning under Support/Query Shift (FSQS) i.e., when support and query instances are sampled from related but different distributions. Our contributions are the following. First, we release a testbed for FSQS, including datasets, relevant baselines and a protocol for a rigorous and reproducible evaluation. Second, we observe that well-established FSL algorithms unsurprisingly suffer from a considerable drop in accuracy when facing FSQS, stressing the significance of our study. Finally, we show that transductive algorithms can limit the inopportune effect of DS. In particular, we study both the role of Batch-Normalization and Optimal Transport (OT) in aligning distributions, bridging Unsupervised Domain Adaptation with FSL. This results in a new method that efficiently combines OT with the celebrated Prototypical Networks. We bring compelling experiments demonstrating the advantage of our method. Our work opens an exciting line of research by providing a testbed and strong baselines. Our code is available at https://github.com/ebennequin/meta-domain-shift. Preprint. Under review at ECML-PKDD 2021.

show abstract

Section: Discussionmentioning

confidence: 99%

Bridging Few-Shot Learning and Adaptation: New Challenges of Support-Query Shift

Bennequin¹,

Bouvier²,

Tami³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The main idea of this paper is to incorporate an appropriate regularization term to learn a classifier f for target domain. Specifically, we attempt to discard the domain alignment, since misalignment tends to do more harm than good in PDA [11] and even in UDA [22]. Motivated by the theoretical analysis, we turn to focus on the model smoothness.…”

Section: Main Ideamentioning

confidence: 99%

“…However, the target samples might be wrongly predicted into the irrelevant categories in the beginning under the domain shift, as shown in Fig. 1 (b) [18], [19], [20], [21], [22], making it hard to obtain a perfect alignment. The experimental results of these methods also confirm that it is difficult to accurately identify the irrelevant categories [11], [12], [17] in source domain.…”

Section: Introductionmentioning

confidence: 99%

“…The experimental results of these methods also confirm that it is difficult to accurately identify the irrelevant categories [11], [12], [17] in source domain. Further, even if the irrelevant source categories are correctly filtered, it is still hard to obtain a perfect alignment to guarantee that the target domain can be labeled correctly [22], [23], as shown in Fig. 1 (c).…”

Section: Introductionmentioning

confidence: 99%

“…1 (c). From a theoretical perspective, several recently proposed theoretical works reveal that the domain alignment hurts the transfer-ability of representations/model [21], [22]. Consequently, these drawbacks push us to seek an alternative for solving PDA without domain alignment.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Partial Domain Adaptation without Domain Alignment

Li¹,

Chen²

2021

Preprint

View full text Add to dashboard Cite

Unsupervised domain adaptation (UDA) aims to transfer knowledge from a well-labeled source domain to a different but related unlabeled target domain with identical label space. Currently, the main workhorse for solving UDA is domain alignment, which has proven successful. However, it is often difficult to find an appropriate source domain with identical label space. A more practical scenario is so-called partial domain adaptation (PDA) in which the source label set or space subsumes the target one. Unfortunately, in PDA, due to the existence of the irrelevant categories in source domain, it is quite hard to obtain a perfect alignment, thus resulting in mode collapse and negative transfer. Although several efforts have been made by down-weighting the irrelevant source categories, the strategies used tend to be burdensome and risky since exactly which irrelevant categories are unknown. These challenges motivate us to find a relatively simpler alternative to solve PDA. To achieve this, we first provide a thorough theoretical analysis, which illustrates that the target risk is bounded by both model smoothness and between-domain discrepancy. Considering the difficulty of perfect alignment in solving PDA, we turn to focus on the model smoothness while discard the riskier domain alignment to enhance the adaptability of the model. Specifically, we instantiate the model smoothness as a quite simple intra-domain structure preserving (IDSP). To our best knowledge, this is the first naive attempt to address the PDA without domain alignment. Finally, our empirical results on multiple benchmark datasets demonstrate that IDSP is not only superior to the PDA SOTAs by a significant margin on some benchmarks (e.g., ∼+10% on Cl→Rw and ∼+8% on Ar→Rw ), but also complementary to domain alignment in the standard UDA.

show abstract

Target Consistency for Domain Adaptation: when Robustness meets Transferability

Ouali¹,

Bouvier²,

Tami³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Learning Invariant Representations has been successfully applied for reconciling a source and a target domain for Unsupervised Domain Adaptation. By investigating the robustness of such methods under the prism of the cluster assumption, we bring new evidence that invariance with a low source risk does not guarantee a wellperforming target classifier. More precisely, we show that the cluster assumption is violated in the target domain despite being maintained in the source domain, indicating a lack of robustness of the target classifier. To address this problem, we demonstrate the importance of enforcing the cluster assumption in the target domain, named Target Consistency (TC), especially when paired with Class-Level InVariance (CLIV). Our new approach results in a significant improvement, on both image classification and segmentation benchmarks, over state-of-the-art methods based on invariant representations. Importantly, our method is flexible and easy to implement, making it a complementary technique to existing approaches for improving transferability of representations. * Equal contribution Preprint. Under review.

show abstract

Robust Domain Adaptation: Representations, Weights and Inductive Bias

Cited by 3 publications

References 17 publications

Bridging Few-Shot Learning and Adaptation: New Challenges of Support-Query Shift

Bridging Few-Shot Learning and Adaptation: New Challenges of Support-Query Shift

Partial Domain Adaptation without Domain Alignment

Target Consistency for Domain Adaptation: when Robustness meets Transferability

Contact Info

Product

Resources

About