Validating Bayesian Inference Algorithms with Simulation-Based Calibration

Abstract

Verifying the correctness of Bayesian computation is challenging. This is especially true for complex models that are common in practice, as these require sophisticated model implementations and algorithms. In this paper we introduce simulation-based calibration (SBC), a general procedure for validating inferences from Bayesian algorithms capable of generating posterior samples. This procedure not only identifies inaccurate computation and inconsistencies in model implementations but also provides graphical summaries that can indicate the nature of the problems that arise. We argue that SBC is a critical part of a robust Bayesian workflow, as well as being a useful tool for those developing computational algorithms and statistical software.Conditioning this joint distribution on an observation, ỹ, yields a posterior distribution, π(θ | ỹ) ∝ π(ỹ, θ),

A Trust Crisis In Simulation-Based Inference? Your Posterior Approximations Can Be Unfaithful

“…This issue is not limited to coverage. Simulation-Based Calibration (sbc) (Talts et al, 2018) relies on samples of arbitrary posterior approximations. Diagnosing nonamortized estimators with sbc therefore requires a similar approach as we have taken in our coverage analyses.…”

“…The mcmc literature deals with this exact same problem in the form of determining whether a set of Markov chain samples have converged to the target distribution (Lin, 2014;Hogg and Foreman-Mackey, 2018). In this regard, empirical diagnostic tools have been proposed over the years (Geweke et al, 1991;Gelman and Rubin, 1992;Raftery and Lewis, 1991;Dixit and Roy, 2017;Talts et al, 2018) and have helped practitioners using mcmc properly. Nonetheless, there is currently no clear solution to determine convergence with absolute certainty (Dixit, 2018;Roy, 2020), even if the likelihood function is here tractable.…”

“…The field, therefore, inherits the quality assessments (Lueckmann et al, 2021) customary to the machine learning literature, such as the minimization of classical divergence criteria. Despite recent developments of post hoc diagnostics to inspect the quality of likelihood-free approximations (Cranmer et al, 2015;Brehmer et al, 2018Brehmer et al, , 2019Hermans et al, 2021;Lueckmann et al, 2021;Talts et al, 2018;Dalmasso et al, 2020), assessing whether approximate inference results are sufficiently reliable for scientific inquiry remains largely unanswered whenever fitting criteria are not globally optimized or whenever the data is limited. In fact, domain sciences, and more specifically the physical sciences, are not necessarily interested in the exactness of an approximation.…”

Refining Epidemiological Forecasts with Simple Scoring Rules

“…Accordingly, the epidemiological community needs a method for assessing forecasts of quantities for which the truth is unknown. Simulation-Based Calibration (SBC) [27] is a candidate method for this task, worth investigating further. There are three worthwhile directions in which we can extend the statistical model presented in Section 2.…”

Toward a Taxonomy of Trust for Probabilistic Machine Learning

“…As we have seen in the case studies, a sufficiently complex Bayesian analysis typically aims to approximate summaries of the posterior. Geweke (2004), Cook et al (2006), andTalts et al (2018) present simulation-based methods for checking the accuracy of a Bayesian approximation algorithm (via its implemented code). We and others have also developed methods for checking approximate computation on a particular dataset of interest; for example, Yao et al (2018) and Huggins et al (2020) have presented methods for evaluating variational inference.…”