Exact and robust conformal inference methods for predictive machine learning with dependent data

Chernozhukov, Victor; Wüthrich, Kaspar; Zhu, Yinchu

doi:10.1920/wp.cem.2018.1618

Cited by 44 publications

(51 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The result is that their prediction intervals -unlike ours -do not become tight, even in the limit. Chernozhukov et al [2018] consider the problem of conformal prediction for time series data, for which the exchangeability assumption may not hold. They show that if the data comes from a rapidly mixing process (so that, in particular, points that are well separated in the sequence are approximately independent) then it is still possible to obtain approximate marginal coverage guarantees.…”

Section: Additional Related Workmentioning

confidence: 99%

Online Multivalid Learning: Means, Moments, and Prediction Intervals

Gupta¹,

Jung²,

Noarov³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present a general, efficient technique for providing contextual predictions that are "multivalid" in various senses, against an online sequence of adversarially chosen examples (x, y). This means that the resulting estimates correctly predict various statistics of the labels y not just marginally -as averaged over the sequence of examples -but also conditionally on x ∈ G for any G belonging to an arbitrary intersecting collection of groups G.We provide three instantiations of this framework. The first is mean prediction, which corresponds to an online algorithm satisfying the notion of multicalibration from Hébert-Johnson et al. [2018]. The second is variance and higher moment prediction, which corresponds to an online algorithm satisfying the notion of mean-conditioned moment multicalibration from Jung et al. [2020]. Finally, we define a new notion of prediction interval multivalidity, and give an algorithm for finding prediction intervals which satisfy it. Because our algorithms handle adversarially chosen examples, they can equally well be used to predict statistics of the residuals of arbitrary point prediction methods, giving rise to very general techniques for quantifying the uncertainty of predictions of black box algorithms, even in an online adversarial setting. When instantiated for prediction intervals, this solves a similar problem as conformal prediction, but in an adversarial environment and with multivalidity guarantees stronger than simple marginal coverage guarantees.

show abstract

Section: Additional Related Workmentioning

confidence: 99%

Online Multivalid Learning: Means, Moments, and Prediction Intervals

Gupta¹,

Jung²,

Noarov³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…(Kuchibhotla, 2020) explicitly describes conformal regions for arbitrary spaces, delivering theoretical results to support the the validity of conformal prediction regions in R d . Chernozhukov et al (2018) provides valid conformal prediction regions for potentially dependent data, generalizing the traditional conformal inference assumption of exchangeability.…”

Section: Joint Prediction Regionsmentioning

confidence: 99%

Conformal Uncertainty Sets for Robust Optimization

Johnstone,

Cox

2021

Preprint

View full text Add to dashboard Cite

Decision-making under uncertainty is hugely important for any decisions sensitive to perturbations in observed data. One method of incorporating uncertainty into making optimal decisions is through robust optimization, which minimizes the worst-case scenario over some uncertainty set. We explore Mahalanobis distance as a novel function for multi-target regression and the construction of joint prediction regions. We also connect conformal prediction regions to robust optimization, providing finite sample valid and conservative uncertainty sets, aptly named conformal uncertainty sets. We compare the coverage and efficiency of the conformal prediction regions generated with Mahalanobis distance to other conformal prediction regions. We also construct a small robust optimization example to compare conformal uncertainty sets to those constructed under the assumption of normality.

show abstract

“…Conformal prediction is a versatile and simple method introduced in [26,24] that provides a finite sample and distribution free 100(1 − α)% confidence region on the predicted object based on past observations. It has been applied for designing uncertainty sets in active learning [12], anomaly detection [14,2], few shot learning [8], time series [5,27,6] or to infer the performance guarantee for statistical learning algorithms [13,4]. We refer to the extensive reviews in [1] for other applications to artificial intelligence.…”

Section: Introductionmentioning

confidence: 99%

Stable Conformal Prediction Sets

Ndiaye¹

2021

Preprint

View full text Add to dashboard Cite

When one observes a sequence of variables (x1, y1), ..., (xn, yn), conformal prediction is a methodology that allows to estimate a confidence set for yn+1 given xn+1 by merely assuming that the distribution of the data is exchangeable. While appealing, the computation of such set turns out to be infeasible in general, e.g., when the unknown variable yn+1 is continuous. In this paper, we combine conformal prediction techniques with algorithmic stability bounds to derive a prediction set computable with a single model fit. We perform some numerical experiments that illustrate the tightness of our estimation when the sample size is sufficiently large.

show abstract

Exact and robust conformal inference methods for predictive machine learning with dependent data

Cited by 44 publications

References 16 publications

Online Multivalid Learning: Means, Moments, and Prediction Intervals

Online Multivalid Learning: Means, Moments, and Prediction Intervals

Conformal Uncertainty Sets for Robust Optimization

Stable Conformal Prediction Sets

Contact Info

Product

Resources

About