Automatic Differentiation Variational Inference with Mixtures

Morningstar, Warren R.; Vikram, Sharad; Ham, Cusuh; Gallagher, Andrew; Dillon, Joshua V.

doi:10.48550/arxiv.2003.01687

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…7 demonstrates. Mixtures have proven difficult to fit in general [Morningstar et al, 2020]. The PAC m family of risks subsume classic risks and offer theoretical generalization guarantees in cases of model misspecification, while remaining computationally tractable.…”

Section: Related Workmentioning

confidence: 99%

PAC$^m$-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regime

Morningstar¹,

Alemi²,

Dillon³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

While the decision-theoretic optimality of the Bayesian formalism under correct model specification is well-known [Berger, 2013], the Bayesian case becomes less clear under model misspecification [Grünwald et al., 2017, Ramamoorthi et al., 2015, Fushiki et al., 2005. To formally understand the consequences of Bayesian misspecification, this work examines the relationship between posterior predictive risk and its sensitivity to correct model assumptions, i.e., choice of likelihood and prior. We present the multisample PAC m -Bayes risk. This risk is justified by theoretical analysis based on PAC-Bayes as well as empirical study on a number of toy problems. The PAC m -Bayes risk is appealing in that it entails direct minimization of the Monte-Carlo approximated posterior predictive risk yet recovers both the Bayesian formalism as well as the MLE in its limits. Our work is heavily influenced by Masegosa [2019]; our contributions are to align training and generalization risks while offering a tighter bound which empirically performs at least as well and sometimes much better.

show abstract

Section: Related Workmentioning

confidence: 99%