A Trust Crisis In Simulation-Based Inference? Your Posterior Approximations Can Be Unfaithful

Hermans, Joeri; Delaunoy, Arnaud; Rozet, François; Wehenkel, Antoine; Begy, Volodimir; Louppe, Gilles

doi:10.48550/arxiv.2110.06581

Cited by 29 publications

(27 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…But the uncertainty estimated by the emulator should be greater than the phenomenological model because of the limited training data. The problem of the underestimated uncertainty may come from the nature of the NF emulator [57]. NF is a series of continuous transformations so has relatively bad performance on learning truncation property.…”

Section: Discussionmentioning

confidence: 99%

Testing the robustness of simulation-based gravitational-wave population inference

Cheung,

Wong,

Hannuksela

et al. 2021

Preprint

View full text Add to dashboard Cite

Gravitational-wave population studies have become more important in gravitational-wave astronomy because of the rapid growth of the observed catalog. In recent studies, emulators based on different machine learning techniques are used to emulate the outcomes of the population synthesis simulation with fast speed. In this study, we benchmark the performance of two emulators that learn the truncated power-law phenomenological model by using Gaussian process regression and normalizing flows techniques to see which one is a more capable likelihood emulator in the population inference. We benchmark the characteristic of the emulators by comparing their performance in the population inference to the phenomenological model using mock and real observation data. Our results suggest that the normalizing flows emulator can recover the posterior distribution by using the phenomenological model in the population inference with up to 300 mock injections. The normalizing flows emulator also underestimates the uncertainty for some posterior distributions in the population inference on real observation data. On the other hand, the Gaussian process regression emulator has poor performance on the same task and can only be used effectively in low-dimension cases.

show abstract

Section: Discussionmentioning

confidence: 99%

Testing the robustness of simulation-based gravitational-wave population inference

Cheung,

Wong,

Hannuksela

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Training parametrized classifiers and density estimators is a challenging task in machine learning, and often requires a large number of simulations and/or a post-hoc calibration procedure in order to produce satisfactory results. It was recently demonstrated that typically-used SBI algorithms tend to produce overly confident posterior estimates [143]an unacceptable outcome in cosmological applications. It is therefore imperative to perform diagnostic coverage tests (as done in, e.g., Refs.…”

Section: Uncertainty Quantification and Biasmentioning

confidence: 99%

Machine Learning and Cosmology

Dvorkin,

Mishra-Sharma,

Nord

et al. 2022

Preprint

View full text Add to dashboard Cite

Methods based on machine learning have recently made substantial inroads in many corners of cosmology. Through this process, new computational tools, new perspectives on data collection, model development, analysis, and discovery, as well as new communities and educational pathways have emerged. Despite rapid progress, substantial potential at the intersection of cosmology and machine learning remains untapped. In this white paper, we summarize current and ongoing developments relating to the application of machine learning within cosmology and provide a set of recommendations aimed at maximizing the scientific impact of these burgeoning tools over the coming decade through both technical development as well as the fostering of emerging communities.

show abstract

“…Following Ref. [18], the expected coverage probability of the 1 − α highest posterior density region (HPDR) of some estimated posterior p(ϑ|x) is given by…”

Section: "Getting It Right" Through Coverage Tests For Simulation-bas...mentioning

confidence: 99%

“…This quantity can be interpreted as both expected Bayesian credibility as well as expected Frequentist coverage probability of p(ϑ|x) [18]. It enables us to estimate the actual error rate α of a 1 − α highest posterior density region of some estimated posterior p(ϑ|x).…”

Section: "Getting It Right" Through Coverage Tests For Simulation-bas...mentioning

confidence: 99%

“…2.3), the trained network has learned not only the marginal posteriors for our observed data, but also the marginal posteriors for any data drawn from the (truncated) prior. This enables us to efficiently inspect the statistical rigor of our results, and assess fundamental statistical properties like expected coverage [18]. These sorts of tests are not feasible for results generated via sampling-based methods, since for each new observation a new chain is needed.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Fast and Credible Likelihood-Free Cosmology with Truncated Marginal Neural Ratio Estimation

Cole,

Miller,

Witte

et al. 2021

Preprint

View full text Add to dashboard Cite

Sampling-based inference techniques are central to modern cosmological data analysis; these methods, however, scale poorly with dimensionality and typically require approximate or intractable likelihoods. In this paper we describe how Truncated Marginal Neural Ratio Estimation (tmnre) (a new approach in so-called simulation-based inference) naturally evades these issues, improving the (i) efficiency, (ii) scalability, and (iii) trustworthiness of the inference. Using measurements of the Cosmic Microwave Background (CMB), we show that tmnre can achieve converged posteriors using orders of magnitude fewer simulator calls than conventional Markov Chain Monte Carlo (mcmc) methods. Remarkably, the required number of samples is effectively independent of the number of nuisance parameters. In addition, a property called local amortization allows the performance of rigorous statistical consistency checks that are not accessible to sampling-based methods. tmnre promises to become a powerful tool for cosmological data analysis, particularly in the context of extended cosmologies, where the timescale required for conventional sampling-based inference methods to converge can greatly exceed that of simple cosmological models such as ΛCDM. To perform these computations, we use an implementation of tmnre via the open-source code swyft. ‡ *

show abstract

A Trust Crisis In Simulation-Based Inference? Your Posterior Approximations Can Be Unfaithful

Cited by 29 publications

References 20 publications

Testing the robustness of simulation-based gravitational-wave population inference

Testing the robustness of simulation-based gravitational-wave population inference

Machine Learning and Cosmology

Fast and Credible Likelihood-Free Cosmology with Truncated Marginal Neural Ratio Estimation

Contact Info

Product

Resources

About