On the Brittleness of Bayesian Inference

Abstract

Abstract. With the advent of high-performance computing, Bayesian methods are becoming increasingly popular tools for the quantification of uncertainty throughout science and industry. Since these methods can impact the making of sometimes critical decisions in increasingly complicated contexts, the sensitivity of their posterior conclusions with respect to the underlying models and prior beliefs is a pressing question to which there currently exist positive and negative answers. We report new results suggesting that, although Bayesian methods are robust when the number of possible outcomes is finite or when only a finite number of marginals of the data-generating distribution are unknown, they could be generically brittle when applied to continuous systems (and their discretizations) with finite information on the data-generating distribution. If closeness is defined in terms of the total variation (TV) metric or the matching of a finite system of generalized moments, then (1) two practitioners who use arbitrarily close models and observe the same (possibly arbitrarily large amount of) data may reach opposite conclusions; and (2) any given prior and model can be slightly perturbed to achieve any desired posterior conclusion. The mechanism causing brittleness/robustness suggests that learning and robustness are antagonistic requirements, which raises the possibility of a missing stability condition when using Bayesian inference in a continuous world under finite information.

Gaussian approximation of general non-parametric posterior distributions

“…We point out that, different from the bounded Lipschitz metric [7,8], our approximation results hold under (stronger) total variation metric that also applies to L 2 credible balls and point-wise credible intervals. Our result can be viewed as complementary to [33,36,34,35] who showed that although Bayesian methods are robust with finite information, they could be brittle when handling continuous systems. Rather, our positive results rely on the facts that the statistical models in consideration are correctly specified and the assigned priors charge the function space (with proper topological and geometrical details) with full mass.…”

Well‐posedness of Bayesian inverse problems in quasi‐Banach spaces with stable priors

“…there is no parameter value in U that corresponds to the 'truth'). Of particular note in this setting is the brittleness phenomenon highlighted by [11,12]: not only does µ y depend upon µ 0 , as would be expected, but it can do so in a highly discontinuous way, in the sense that any pre-specified quantity of interest can be made to have any desired posterior expectation value after arbitrarily small perturbation in the common-moments, weak, total variation, or Hellinger topologies. This brittleness phenomenon takes place in the limit as the data resolution becomes infinitely fine, and it is a topic of current research whether approaches such as coarsening can in general yield robust inferences [10].…”

Gamblets for opening the complexity-bottleneck of implicit schemes for hyperbolic and parabolic ODEs/PDEs with rough coefficients

“…5] of the linearity of the PDE and the quadratic nature of the loss function. For non linear PDEs or non quadratic loss functions, although optimal priors (which may not be Gaussian) could in principle be numerically approximated, such approximations could be severely impacted by stability issues as discussed in [67,63,68,64]. , Ω = (0, 1) 2 and T h is a square grid of mesh size h = (1 + 2 q ) −1 with r = 6 and 64 × 64 interior nodes, a is piecewise constant on each square of T h and given by a(x) = Π r k=1 1+0.5 cos(2 k π( Figure 2.…”