PAC-Bayes and domain adaptation

Germain, Pascal; Habrard, Amaury; Laviolette, François; Morvant, Emilie

doi:10.1016/j.neucom.2019.10.105

Cited by 31 publications

(48 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Pioneering theoretical work was proposed by Ben-David et al [10], which shows that the target risk is upper bounded by three terms: source risk, marginal distribution discrepancy, and combined risk. This learning bound has been extended from many perspectives, such as considering different loss functions [32], different distribution distances [33], [34], [35] or the PAC-Bayes framework [36], [37]. According to the survey [14], most works focus on proving tighter bounds by constructing a new distribution distance.…”

Section: Domain Adaptation Theorymentioning

confidence: 99%

Semi-Supervised Heterogeneous Domain Adaptation: Theory and Algorithms

Fang

Liu

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Semi-supervised heterogeneous domain adaptation (SsHeDA) aims to train a classifier for the target domain, in which only unlabeled and a small number of labeled data are available. This is done by leveraging knowledge acquired from a heterogeneous source domain. From algorithmic perspectives, several methods have been proposed to solve the SsHeDA problem; yet there is still no theoretical foundation to explain the nature of the SsHeDA problem or to guide new and better solutions. Motivated by compatibility condition in semi-supervised probably approximately correct (PAC) theory, we explain the SsHeDA problem by proving its generalization error -that is, why labeled heterogeneous source data and unlabeled target data help to reduce the target risk. Guided by our theory, we devise two algorithms as proof of concept. One, kernel heterogeneous domain alignment (KHDA), is a kernel-based algorithm; the other, joint mean embedding alignment (JMEA), is a neural network-based algorithm. When a dataset is small, KHDA's training time is less than JMEA's. When a dataset is large, JMEA is more accurate in the target domain. Comprehensive experiments with image/text classification tasks show KHDA to be the most accurate among all non-neural network baselines, and JMEA to be the most accurate among all baselines.

show abstract

Section: Domain Adaptation Theorymentioning

confidence: 99%

Semi-Supervised Heterogeneous Domain Adaptation: Theory and Algorithms

Fang

Liu

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…An extension of hypothesis transfer learning is studied in [42], where an algorithm combining the hypotheses from multiple sources based on regularized ERM principle is studied. There are also works focusing on the theoretical aspects of domain adaptation, see [5,11,43,44,45,46], which are also related to our problem. Note that in domain adaptation, there is no labeled target data and only unlabeled target samples are available.…”

Section: Related Workmentioning

confidence: 99%

Characterizing and Understanding the Generalization Error of Transfer Learning with Gibbs Algorithm

Bu¹,

Aminian²,

Toni³

et al. 2021

Preprint

View full text Add to dashboard Cite

We provide an information-theoretic analysis of the generalization ability of Gibbs-based transfer learning algorithms by focusing on two popular transfer learning approaches, α-weighted-ERM and two-stage-ERM.Our key result is an exact characterization of the generalization behaviour using the conditional symmetrized KL information between the output hypothesis and the target training samples given the source samples. Our results can also be applied to provide novel distribution-free generalization error upper bounds on these two aforementioned Gibbs algorithms. Our approach is versatile, as it also characterizes the generalization errors and excess risks of these two Gibbs algorithms in the asymptotic regime, where they converge to the α-weighted-ERM and two-stage-ERM, respectively. Based on our theoretical results, we show that the benefits of transfer learning can be viewed as a bias-variance trade-off, with the bias induced by the source distribution and the variance induced by the lack of target samples. We believe this viewpoint can guide the choice of transfer learning algorithms in practice.

show abstract

“…PAC-Bayesian analysis of distribution shift. Performance under distribution shift has also been characterized under the PAC-Bayesian setting where the learning algorithm outputs a posterior distribution over the h hypothesis class [37,38,61]. Li and Bilmes [61] directly bound the error on the target distribution (OOD) in terms of the empirical error on a small number of labeled samples from the target and a "divergence prior" which measures some divergence between the source and target domains.…”

Section: F Additional Related Workmentioning

confidence: 99%

Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization

Miller,

Taori,

Raghunathan

et al. 2021

Preprint

View full text Add to dashboard Cite

For machine learning systems to be reliable, we must understand their performance in unseen, out-of-distribution environments. In this paper, we empirically show that out-of-distribution performance is strongly correlated with in-distribution performance for a wide range of models and distribution shifts. Specifically, we demonstrate strong correlations between in-distribution and out-of-distribution performance on variants of CIFAR-10 & ImageNet, a synthetic pose estimation task derived from YCB objects, satellite imagery classification in FMoW-WILDS, and wildlife classification in iWildCam-WILDS. The strong correlations hold across model architectures, hyperparameters, training set size, and training duration, and are more precise than what is expected from existing domain adaptation theory. To complete the picture, we also investigate cases where the correlation is weaker, for instance some synthetic distribution shifts from CIFAR-10-C and the tissue classification dataset Camelyon17-WILDS. Finally, we provide a candidate theory based on a Gaussian data model that shows how changes in the data covariance arising from distribution shift can affect the observed correlations.

show abstract

PAC-Bayes and domain adaptation

Cited by 31 publications

References 32 publications

Semi-Supervised Heterogeneous Domain Adaptation: Theory and Algorithms

Semi-Supervised Heterogeneous Domain Adaptation: Theory and Algorithms

Characterizing and Understanding the Generalization Error of Transfer Learning with Gibbs Algorithm

Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization

Contact Info

Product

Resources

About