PAC$^m$-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regime

Abstract

While the decision-theoretic optimality of the Bayesian formalism under correct model specification is well-known [Berger, 2013], the Bayesian case becomes less clear under model misspecification [Grünwald et al., 2017, Ramamoorthi et al., 2015, Fushiki et al., 2005. To formally understand the consequences of Bayesian misspecification, this work examines the relationship between posterior predictive risk and its sensitivity to correct model assumptions, i.e., choice of likelihood and prior. We present the multisample PAC m -Bayes risk. This risk is justified by theoretical analysis based on PAC-Bayes as well as empirical study on a number of toy problems. The PAC m -Bayes risk is appealing in that it entails direct minimization of the Monte-Carlo approximated posterior predictive risk yet recovers both the Bayesian formalism as well as the MLE in its limits. Our work is heavily influenced by Masegosa [2019]; our contributions are to align training and generalization risks while offering a tighter bound which empirically performs at least as well and sometimes much better.

Robust PAC$^m$: Training Ensemble Models Under Model Misspecification and Outliers

“…The underlying in-distribution (ID) measure ν(x) (dashed line -a mixture of Gaussians) produces the data points in black, while the contaminating out-ofdistribution (OOD) measure ξ(x) produces the data point in red. The ensemble predictive distributions minimizing the conventional energy free criterion is denoted as J ; the one minimizing the m-free energy of [7] with m = 10 by J 10 ; and the ones minimizing the proposed robust m-free energy with m = 10 and t = {0.9, 0.7, 0.5} by J 10 0.9 , J 10 0.7 , and J 10 0.5 , respectively.…”

“…Recent work has addressed the problem of model misspecification for Bayesian learning. In particular, references [7], [8] have argued that the minimization of the standard free energy criterion yields predictive distributions that do not take advantage of ensembling, and thus have poor generalization capabilities for misspecified models. To illustrate this issue, consider the example in Figure 1.…”

“…To mitigate this problem, references [7], [8] introduced alternative free energy criteria that account for misspecification. The author of [8] leveraged a second-order Jensen's inequality to obtain a tighter bound on the cross entropy loss; while the work [7] proposed an m-free energy criterion that accounts for the performance of an ensemble predictor with m constituent models.…”

“…To mitigate this problem, references [7], [8] introduced alternative free energy criteria that account for misspecification. The author of [8] leveraged a second-order Jensen's inequality to obtain a tighter bound on the cross entropy loss; while the work [7] proposed an m-free energy criterion that accounts for the performance of an ensemble predictor with m constituent models. Both optimization criteria were shown to be effective in overcoming the shortcomings of standard Bayesian learning under misspecification, by yielding posteriors that make better use of ensembling.…”

Priors in Bayesian Deep Learning: A Review

“…In a similar vein, Savage's theorem [25], which promises us optimal decisions under Bayesian decision theory, breaks down under prior misspecification [26]. Finally, it can even be shown that PAC-Bayesian inference can exceed the Bayesian one in terms of generalization performance when the prior is misspecified [27,28].…”