Distribution-free, Risk-controlling Prediction Sets

Bates, Stephen; Angelopoulos, Anastasios N.; Lei, Lihua; Malik, Jitendra; Jordan, Michael I.

doi:10.1145/3478535

Cited by 68 publications

(77 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the proof of this fact, along with a discussion of the tighter confidence bounds used in the experiments, see previous work [4,13]. This calibration procedure is easy to implement in code, and we summarize it in Algorithm 2.…”

Section: Algorithm 2 Pseudocode For Computing λmentioning

confidence: 87%

“…Definition 1 (Risk-Controlling Prediction Set (RCPS), modified from [4]). We call a random set-valued function…”

Section: Methodsmentioning

confidence: 99%

“…Of particular interest to us is the method of conformalized quantile regression (CQR) [6]. We directly build on CQR, by replacing the the conformal subroutine with the fixed-sequence testing procedure from [4,13]. Other works have applied distribution-free uncertainty quantification to biological and medical computer vision tasks [76][77][78][79][80][81].…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging

Angelopoulos¹,

Kohli²,

Bates³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Image-to-image regression is an important learning task, used frequently in biological imaging. Current algorithms, however, do not generally offer statistical guarantees that protect against a model's mistakes and hallucinations. To address this, we develop uncertainty quantification techniques with rigorous statistical guarantees for image-to-image regression problems. In particular, we show how to derive uncertainty intervals around each pixel that are guaranteed to contain the true value with a user-specified confidence probability. Our methods work in conjunction with any base machine learning model, such as a neural network, and endow it with formal mathematical guarantees-regardless of the true unknown data distribution or choice of model. Furthermore, they are simple to implement and computationally inexpensive. We evaluate our procedure on three image-to-image regression tasks: quantitative phase microscopy, accelerated magnetic resonance imaging, and super-resolution transmission electron microscopy of a Drosophila melanogaster brain.

show abstract

Section: Algorithm 2 Pseudocode For Computing λmentioning

confidence: 87%

“…Definition 1 (Risk-Controlling Prediction Set (RCPS), modified from [4]). We call a random set-valued function…”

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging

Angelopoulos¹,

Kohli²,

Bates³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The experiments in section 5 use the Conformal Prediction (CP) framework (Vovk et al, 2005). While recent work found CP to be useful in many practical applications such as medical image segmentation (Bates et al, 2021) and adaptive computation Transformers , one of CP challenges is in reducing the size of the prediction sets while maintaining the compelling accuracy guarantees. In this work, we follow the expanded admission CP extension of that leverages the existence of equally correct answers to improve the statistical efficiency of the calibration.…”

Section: Ethical Considerationsmentioning

confidence: 99%

Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation

Bulian¹,

Buck²,

Gajewski³

et al. 2022

Preprint

View full text Add to dashboard Cite

The predictions of question answering (QA) systems are typically evaluated against manually annotated finite sets of one or more answers. This leads to a coverage limitation that results in underestimating the true performance of systems, and is typically addressed by extending over exact match (EM) with predefined rules or with the token-level F 1 measure. In this paper, we present the first systematic conceptual and data-driven analysis to examine the shortcomings of token-level equivalence measures. To this end, we define the asymmetric notion of answer equivalence (AE), accepting answers that are equivalent to or improve over the reference, and collect over 26K human judgements for candidates produced by multiple QA systems on SQuAD. 1 Through a careful analysis of this data, we reveal and quantify several concrete limitations of the F 1 measure, such as false impression of graduality, missing dependence on question, and more. Since collecting AE annotations for each evaluated model is expensive, we learn a BERT matching (BEM) measure to approximate this task. Being a simpler task than QA, we find BEM to provide significantly better AE approximations than F 1 , and more accurately reflect the performance of systems. Finally, we also demonstrate the practical utility of AE and BEM on the concrete application of minimal accurate prediction sets, reducing the number of required answers by up to ×2.6.

show abstract

“…Our work builds upon prior literature on distribution-free uncertainty quantification, which we review in-depth in Appendix A. Within this literature, the work most closely related to ours is arguably the work by Bates et al [15], which has focused on generating set-valued predictions from a black-box predictor that controls the expected loss on future test points at a user-specified level. While one can view our problem from the perspective of set-valued predictions, applying their methodology to find near-optimal solutions in our problem is not straightforward, and one would need to assume to have access to qualification labels for all the candidates in multiple pools, something we view as rather impractical.…”

Section: Introductionmentioning

confidence: 99%

Improving Screening Processes via Calibrated Subset Selection

Wang¹,

Joachims²,

Gomez-Rodriguez³

2022

Preprint

View full text Add to dashboard Cite

Many selection processes such as finding patients qualifying for a medical trial or retrieval pipelines in search engines consist of multiple stages, where an initial screening stage focuses the resources on shortlisting the most promising candidates. In this paper, we investigate what guarantees a screening classifier can provide, independently of whether it is constructed manually or trained. We find that current solutions do not enjoy distribution-free theoretical guarantees-we show that, in general, even for a perfectly calibrated classifier, there always exist specific pools of candidates for which its shortlist is suboptimal. Then, we develop a distribution-free screening algorithm-called Calibrated Subset Selection (CSS)-that, given any classifier and some amount of calibration data, finds near-optimal shortlists of candidates that contain a desired number of qualified candidates in expectation. Moreover, we show that a variant of our algorithm that calibrates a given classifier multiple times across specific groups can create shortlists with provable diversity guarantees. Experiments on US Census survey data validate our theoretical results and show that the shortlists provided by our algorithm are superior to those provided by several competitive baselines.

show abstract

Distribution-free, Risk-controlling Prediction Sets

Cited by 68 publications

References 34 publications

Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging

Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging

Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation

Improving Screening Processes via Calibrated Subset Selection

Contact Info

Product

Resources

About