Optimally tackling covariate shift in RKHS-based nonparametric regression

Abstract

We study the covariate shift problem in the context of nonparametric regression over a reproducing kernel Hilbert space (RKHS). We focus on two natural families of covariate shift problems defined using the likelihood ratios between the source and target distributions. When the likelihood ratios are uniformly bounded, we prove that the kernel ridge regression (KRR) estimator with a carefully chosen regularization parameter is minimax rate-optimal (up to a log factor) for a large family of RKHSs with regular kernel eigenvalues. Interestingly, KRR does not require full knowledge of the likelihood ratio apart from an upper bound on it. In striking contrast to the standard statistical setting without covariate shift, we also demonstrate that a naïve estimator, which minimizes the empirical risk over the function class, is strictly suboptimal under covariate shift as compared to KRR. We then address the larger class of covariate shift problems where likelihood ratio is possibly unbounded yet has a finite second moment. Here, we show via careful simulations that KRR fails to attain the optimal rate. Instead, we propose a reweighted KRR estimator that weights samples based on a careful truncation of the likelihood ratios. Again, we are able to show that this estimator is minimax optimal, up to logarithmic factors.

Pseudo-Labeling for Kernel Ridge Regression under Covariate Shift

“…Theorem 2 in Ma et al (2022) provides general minimax lower bounds on the excess risk when dQ dP ≤ B and the eigenvalues {µ j } ∞ j=1 are regular, which are covered by our setting. Based on that, direct computation shows that the bounds in Corollary 4.1 are minimax optimal up to logarithmic factors.…”

“…The reweighted empirical risk then serves as the objective function. The weights may be truncated so as to reduce the variance (Shimodaira, 2000;Cortes et al, 2010;Sugiyama and Kawanabe, 2012;Ma et al, 2022). Such methods need absolute continuity of the target distribution with respect to the source, as well as knowledge about the likelihood ratio.…”

“…Lei et al (2021) obtained results for ridge regression in a Bayesian setting. Ma et al (2022) conducted an in-depth analysis of regression in an RKHS when the target covariate distribution and the source one have bounded likelihood ratio or finite χ 2 divergence. In the former case, KRR with a properly chosen penalty was shown to be minimax optimal up to a logarithmic factor.…”

“…• The assumption Σ 0 BΣ can be implied by the "bounded density ratio" assumption (Q P and dQ dP ≤ B) in the study of covariate shift (Cortes et al, 2010;Ma et al, 2022). Aside from that, Σ 0 BΣ does not really require Q P .…”