Testing whether a Learning Procedure is Calibrated

Cockayne, Jon; Graham, Matthew M.; Oates, Chris J.; Sullivan, T. J.; Teymur, Onur

doi:10.48550/arxiv.2012.12670

Cited by 3 publications

(3 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the purpose of this exploratory work, values of Z that are orders of magnitude smaller than 1 are interpreted as indicating that the distributional output from the PNM is under-confident, while values that are orders of magnitude greater than 1 indicate that the PNM is over-confident. A PNM that is neither under nor over confident is said to be calibrated (precise definitions of the term "calibrated" can be found in Cockayne et al, 2021;Karvonen et al, 2020, but the results we present are straight-forward to interpret using the informal approach just described). Our goal in this work is to develop an approximately Bayesian PNM for nonlinear PDEs that is both accurate and calibrated.…”

Section: Experimental Assessmentmentioning

confidence: 93%

Bayesian Numerical Methods for Nonlinear Partial Differential Equations

Wang,

Cockayne,

Chkrebtii

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The numerical solution of differential equations can be formulated as an inference problem to which formal statistical approaches can be applied. However, nonlinear partial differential equations (PDEs) pose substantial challenges from an inferential perspective, most notably the absence of explicit conditioning formula. This paper extends earlier work on linear PDEs to a general class of initial value problems specified by nonlinear PDEs, motivated by problems for which evaluations of the right-hand-side, initial conditions, or boundary conditions of the PDE have a high computational cost. The proposed method can be viewed as exact Bayesian inference under an approximate likelihood, which is based on discretisation of the nonlinear differential operator. Proof-of-concept experimental results demonstrate that meaningful probabilistic uncertainty quantification for the unknown solution of the PDE can be performed, while controlling the number of times the right-handside, initial and boundary conditions are evaluated. A suitable prior model for the solution of the PDE is identified using novel theoretical analysis of the sample path properties of Matérn processes, which may be of independent interest.

show abstract

Section: Experimental Assessmentmentioning

confidence: 93%

Bayesian Numerical Methods for Nonlinear Partial Differential Equations

Wang,

Cockayne,

Chkrebtii

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We review the definition of calibration for probabilistic linear solvers (Definition 17, Lemma 18), discuss the difference between certain random variables (Remark 19), present two illustrations (Examples 20 and 21), and explain why this notion of calibration does not apply to BayesCG under the Krylov prior (Remark 22). Informally, a probabilistic numerical solver is calibrated if its posterior distributions accurately model the uncertainty in the solution [6,7].…”

Section: Calibrationmentioning

confidence: 99%

Statistical Properties of the Probabilistic Numeric Linear Solver BayesCG

Reid¹,

Ipsen²,

Cockayne³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

We analyse the calibration of BayesCG under the Krylov prior, a probabilistic numeric extension of the Conjugate Gradient (CG) method for solving systems of linear equations with symmetric positive definite coefficient matrix. Calibration refers to the statistical quality of the posterior covariances produced by a solver. Since BayesCG is not calibrated in the strict existing notion, we propose instead two test statistics that are necessary but not sufficient for calibration: the Z-statistic and the new S-statistic. We show analytically and experimentally that under low-rank approximate Krylov posteriors,

show abstract

“…If the true quantity of interest q * was genuinely a sample from Q(0, •)|D, then S 2 would follow a χ 2 distribution with |T | degrees of freedom. This observation enables the calibration of a PN method to be assessed [66]. Both metrics naturally require an accurate approximation to q * to act as the ground truth, which is available using brute force computation in Sections 4.1 and 4.2 but not in Section 4.3.…”

Section: Experimental Assessmentmentioning

confidence: 99%

Black Box Probabilistic Numerics

Teymur¹,

Breen²,

Karvonen³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Probabilistic numerics casts numerical tasks, such the numerical solution of differential equations, as inference problems to be solved. One approach is to model the unknown quantity of interest as a random variable, and to constrain this variable using data generated during the course of a traditional numerical method. However, data may be nonlinearly related to the quantity of interest, rendering the proper conditioning of random variables difficult and limiting the range of numerical tasks that can be addressed. Instead, this paper proposes to construct probabilistic numerical methods based only on the final output from a traditional method. A convergent sequence of approximations to the quantity of interest constitute a dataset, from which the limiting quantity of interest can be extrapolated, in a probabilistic analogue of Richardson's deferred approach to the limit. This black box approach (1) massively expands the range of tasks to which probabilistic numerics can be applied, (2) inherits the features and performance of state-of-the-art numerical methods, and (3) enables provably higher orders of convergence to be achieved. Applications are presented for nonlinear ordinary and partial differential equations, as well as for eigenvalue problems-a setting for which no probabilistic numerical methods have yet been developed.

show abstract

Testing whether a Learning Procedure is Calibrated

Cited by 3 publications

References 32 publications

Bayesian Numerical Methods for Nonlinear Partial Differential Equations

Bayesian Numerical Methods for Nonlinear Partial Differential Equations

Statistical Properties of the Probabilistic Numeric Linear Solver BayesCG

Black Box Probabilistic Numerics

Contact Info

Product

Resources

About