Likelihood-Free Frequentist Inference: Confidence Sets with Correct Conditional Coverage

Dalmasso, Niccolò; Masserano, Luca; Zhao, David; Izbicki, Rafael; Lee, Ann B.

doi:10.48550/arxiv.2107.03920

Cited by 5 publications

(13 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From a Bayesian perspective, Rozet and Louppe (2021) propose using the focal and the peripheral losses to weigh down easily classified samples as a means to tune the conservativeness of a posterior estimator. Dalmasso et al (2021) consider the frequentist setting and introduce a theoreticallygrounded algorithm for the construction of confidence intervals that are guaranteed to have perfect coverage, regardless of the quality of the used statistic. Second, in light of our results that ensembles produce more conservative posteriors, model averaging constitutes another promising direction of study, as a simple and efficient method to produce reliable posterior estimators.…”

Section: Discussionmentioning

confidence: 99%

A Trust Crisis In Simulation-Based Inference? Your Posterior Approximations Can Be Unfaithful

Hermans¹,

Delaunoy²,

Rozet³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present extensive empirical evidence showing that current Bayesian simulation-based inference algorithms are inadequate for the falsificationist methodology of scientific inquiry. Our results collected through months of experimental computations show that all benchmarked algorithms -(s)npe, (s)nre, snl and variants of abc -may produce overconfident posterior approximations, which makes them demonstrably unreliable and dangerous if one's scientific goal is to constrain parameters of interest. We believe that failing to address this issue will lead to a well-founded trust crisis in simulation-based inference. For this reason, we argue that research efforts should now consider theoretical and methodological developments of conservative approximate inference algorithms and present research directions towards this objective. In this regard, we show empirical evidence that ensembles are consistently more reliable.

show abstract

Section: Discussionmentioning

confidence: 99%

A Trust Crisis In Simulation-Based Inference? Your Posterior Approximations Can Be Unfaithful

Hermans¹,

Delaunoy²,

Rozet³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…WALDO expands on the framework formalized in [16], which consists of a modular procedure to (i) estimate a likelihood-based test statistic via odds ratios, (ii) estimate critical values C θ0,α across the parameter space via quantile regression, and (iii) check that the constructed confidence sets achieve the desired coverage level for all θ ∈ Θ. Here, we replace (i) and instead use posteriors or point predictions to compute τ WALDO in (4).…”

Section: Methodsmentioning

confidence: 99%

“…(iii) Coverage diagnostics. To check that the constructed confidence sets indeed achieve the desired level of conditional coverage, we leverage the diagnostics procedure introduced in [16]. For a third simulated train set T = {(θ (j) , D (j) )} B j=1 , we construct a confidence region for each D (j) ∈ T and then regress 1{θ (j) ∈ R(D (j) )} against θ (j) adopting a suitable regression method.…”

Section: Methodsmentioning

confidence: 99%

“…Existing approaches for uncertainty quantification in SBI (see [15] for a recent review) are either based on Bayesian posterior estimation [6,41,45,38,25,11,29,49] or on inversion of likelihood ratio tests [9,8,16]. WALDO differs from the former by aiming for conditional coverage instead of marginal coverage, and from the latter by taking advantage of the power of predictive ML algorithms.…”

Section: Relation To Other Workmentioning

confidence: 99%

“…It does so by exploiting estimates of the conditional mean E[θ|D] and conditional variance V[θ|D], but it is indifferent to the source of these quantities. WALDO stems from a recent likelihood-free frequentist inference (LF2I) framework proposed in [16], which showed how to construct valid confidence regions using likelihood estimates in a SBI setting. However, the authors also demonstrated that the statistical power of the resulting sets could easily degrade due to numerical approximations when using a likelihood-based test statistic.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Simulation-Based Inference with Waldo: Confidence Regions by Leveraging Prediction Algorithms or Posterior Estimators for Inverse Problems

Masserano¹,

Dorigo²,

Izbicki³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

The vast majority of modern machine learning targets prediction problems, with algorithms such as Deep Neural Networks revolutionizing the accuracy of point predictions for high-dimensional complex data. Predictive approaches are now used in many domain sciences to directly estimate internal parameters of interest in theoretical simulator-based models. In parallel, common alternatives focus on estimating the full posterior using modern neural density estimators such as normalizing flows. However, an open problem in simulation-based inference (SBI) is how to construct properly calibrated confidence regions for internal parameters with nominal conditional coverage and high power. Many SBI methods are indeed known to produce overly confident posterior approximations, yielding misleading uncertainty estimates. Similarly, existing approaches for uncertainty quantification in deep learning provide no guarantees on conditional coverage. In this work, we present WALDO, a novel method for constructing correctly calibrated confidence regions in SBI. WALDO reframes the well-known Wald test and uses Neyman inversion to convert point predictions and posteriors from any prediction or posterior estimation algorithm to confidence sets with correct conditional coverage, even for finite sample sizes. As a concrete example, we demonstrate how a recently proposed deep learning prediction approach for particle energies in high-energy physics can be recalibrated using WALDO to produce confidence intervals with correct coverage and high power.Preprint. Under review.

show abstract

Statistical constraints on climate model parameters using a scalable cloud-based inference framework

Carzon

Abreu

Regayre

et al. 2023

Environ. Data Science

View full text Add to dashboard Cite

Atmospheric aerosols influence the Earth’s climate, primarily by affecting cloud formation and scattering visible radiation. However, aerosol-related physical processes in climate simulations are highly uncertain. Constraining these processes could help improve model-based climate predictions. We propose a scalable statistical framework for constraining the parameters of expensive climate models by comparing model outputs with observations. Using the C3.AI Suite, a cloud computing platform, we use a perturbed parameter ensemble of the UKESM1 climate model to efficiently train a surrogate model. A method for estimating a data-driven model discrepancy term is described. The strict bounds method is applied to quantify parametric uncertainty in a principled way. We demonstrate the scalability of this framework with 2 weeks’ worth of simulated aerosol optical depth data over the South Atlantic and Central African region, written from the model every 3 hr and matched in time to twice-daily MODIS satellite observations. When constraining the model using real satellite observations, we establish constraints on combinations of two model parameters using much higher time-resolution outputs from the climate model than previous studies. This result suggests that within the limits imposed by an imperfect climate model, potentially very powerful constraints may be achieved when our framework is scaled to the analysis of more observations and for longer time periods.

show abstract

Likelihood-Free Frequentist Inference: Confidence Sets with Correct Conditional Coverage

Cited by 5 publications

References 52 publications

A Trust Crisis In Simulation-Based Inference? Your Posterior Approximations Can Be Unfaithful

A Trust Crisis In Simulation-Based Inference? Your Posterior Approximations Can Be Unfaithful

Simulation-Based Inference with Waldo: Confidence Regions by Leveraging Prediction Algorithms or Posterior Estimators for Inverse Problems

Statistical constraints on climate model parameters using a scalable cloud-based inference framework

Contact Info

Product

Resources

About