Prediction Sets Adaptive to Unknown Covariate Shift

Abstract

Predicting sets of outcomes-instead of unique outcomes-is a promising solution to uncertainty quantification in statistical learning. Despite a rich literature on constructing prediction sets with statistical guarantees, adapting to unknown covariate shift-a prevalent issue in practice-poses a serious challenge and has yet to be solved. In the framework of semiparametric statistics, we can view the covariate shift as a nuisance parameter. In this paper, we propose a novel flexible distribution-free method, PredSet-1Step, to construct prediction sets that can efficiently adapt to unknown covariate shift. PredSet-1Step relies on a one-step correction of the plug-in estimator of coverage error.We theoretically show that our methods are asymptotically probably approximately correct (PAC), having low coverage error with high confidence for large samples. PredSet-1Step may also be used to construct asymptotically risk-controlling prediction sets. We illustrate that our method has good coverage in a number of experiments and by analyzing a data set concerning HIV risk prediction in a South African cohort study. In experiments without covariate shift, PredSet-1Step performs similarly to inductive conformal prediction, which has finite-sample PAC properties. Thus, PredSet-1Step may be used in the common scenario if the user suspectsbut may not be certain-that covariate shift is present, and does not know the form of the shift. Our theory hinges on a new bound for the convergence rate of Wald confidence interval coverage for general asymptotically linear estimators. This is a technical tool of independent interest.

Joint Coverage Regions: Simultaneous Confidence and Prediction Sets

“…Prediction sets have a rich statistical history dating back to Wilks (1941), Wald (1943), Scheffe and Tukey (1945), and Tukey (1947Tukey ( , 1948. There is an large body of work on constructing prediction sets with coverage guarantees under various assumptions (see, e.g., Bates et al, 2021;Chernozhukov et al, 2018;Dunn et al, 2018;Lei and Wasserman, 2014;Lei et al, 2013Lei et al, , 2015Lei et al, , 2018aPark et al, 2020Park et al, , 2021Sadinle et al, 2019;Kaur et al, 2022;Qiu et al, 2022;Li et al, 2022;Sesia et al, 2022). Among these, one of the best-known methods is conformal prediction (CP) (see, e.g., Vovk et al, 1999;Papadopoulos et al, 2002;Vovk et al, 2022;Chernozhukov et al, 2018;Dunn et al, 2018;Lei and Wasserman, 2014;Lei et al, 2013Lei et al, , 2018a.…”

PAC Prediction Sets for Meta-Learning

“…Thus, these prediction sets cover the labels for most future test examples. Conditional coverage can be extended to hold under covariate shift, assuming the distributions are sufficiently smooth [6] or well estimated [7]. In meta learning, the meta conformal prediction set approach also has a PAC property [14], while losing control over the randomness from the adaptation test examples.…”

“…Prediction sets are a promising approach, since they provide theoretical guarantees when the training and test data are i.i.d. [1,2,3,4]; there have been extensions to handle covariate shift [5,6,7] and online learning [8].…”

Conformal Frequency Estimation with Sketched Data under Relaxed Exchangeability

“…This work builds upon conformal inference, which was pioneered by Vovk and collaborators (Saunders et al, 1999;Vovk et al, 2005) and brought to the statistics spotlight by works such as Lei et al (2013); Lei and Wasserman (2014); Lei et al (2018). Although primarily conceived for supervised prediction (Vovk et al, 2009;Vovk, 2015;Lei and Wasserman, 2014;Romano et al, 2019;Izbicki et al, 2019;Park et al, 2021;Qiu et al, 2022), conformal inference has found other applications including outlier and anomaly detection Kaur et al, 2022;Li et al, 2022), causal inference (Lei et al, 2021, e.g.,), and survival analysis . We mention here that the ideas in conformal prediction have deep roots in statistics, dating back at least to the pioneering works of Wilks (1941), Wald (1943), Scheffe and Tukey (1945), and Tukey (1947, 1948; see also Geisser (2017).…”