Distribution-Free Prediction Sets

Abstract

This paper introduces a new approach to prediction by bringing together two different nonparametric ideas: distribution free inference and nonparametric smoothing. Specifically, we consider the problem of constructing nonparametric tolerance/prediction sets. We start from the general conformal prediction approach and we use a kernel density estimator as a measure of agreement between a sample point and the underlying distribution. The resulting prediction set is shown to be closely related to plug-in density level sets with carefully chosen cut-off values. Under standard smoothness conditions, we get an asymptotic efficiency result that is near optimal for a wide range of function classes. But the coverage is guaranteed whether or not the smoothness conditions hold and regardless of the sample size. The performance of our method is investigated through simulation studies and illustrated in a real data example.

Distribution-free Prediction Bands for Non-parametric Regression

“…We start by recalling the construction of joint prediction sets by using kernel density estimators together with the idea of conformal prediction, as described in Lei et al (2011), using the idea of conformal prediction that was developed in Shafer and Vovk (2008) and Vovk et al (2005Vovk et al ( , 2009. This approach is shown to have finite sample validity as well as asymptotic efficiency under regularity conditions.…”

“…A straightforward approach is to apply the method that was developed in Lei et al (2011) to P j ≡ L.X, Y |X ∈ A j /, the joint distribution of .X, Y/ conditional on the event X ∈ A j . Note that we are mostly interested in the case max j diam.A j / → 0; therefore the marginal density of X within P j becomes increasingly close to uniform.…”

Criteria of Efficiency for Conformal Prediction

“…This idealised conformity measure was introduced by an anonymous referee of the conference version of [3], but its non-idealised analogue in the case of regression had been used in [11] (following [10] and literature on minimum volume prediction). We say that an idealised conformity measure A is a refinement of an idealised conformity measure B if…”

An exact and robust conformal inference method for counterfactual and synthetic controls

“…Conceptually, our procedure builds on the literature on conformal prediction (Vovk et al, 2005(Vovk et al, , 2009; Lei et al, 2013Lei et al, , 2017 and, more broadly, on the literature on permutation tests (Romano, 1990;Lehmann and Romano, 2005), which was started by Fisher (1935) in the context of randomization; see also Rubin (1984) for a Bayesian justification. Conformal inference, a form of permutation inference, is a distribution-free approach for forming prediction intervals.…”