On the Robustness to Misspecification of $α$-Posteriors and Their Variational Approximations

Medina, Marco Avella; Olea, José Luis Montiel; Rush, Cynthia; Velez, Amilcar

doi:10.48550/arxiv.2104.08324

Cited by 2 publications

(2 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More theoretical studies on variational inference (using PAC-Bayes, or not) appeared at the same time or since: [50,93,122,48,145,51,20,19,138,69].…”

Section: Variational Approximationsmentioning

confidence: 99%

User-friendly introduction to PAC-Bayes bounds

Alquier¹

2021

Preprint

View full text Add to dashboard Cite

Aggregated predictors are obtained by making a set of basic predictors vote according to some weights, that is, to some probability distribution.Randomized predictors are obtained by sampling in a set of basic predictors, according to some prescribed probability distribution.Thus, aggregated and randomized predictors have in common that they are not defined by a minimization problem, but by a probability distribution on the set of predictors. In statistical learning theory, there is a set of tools designed to understand the generalization ability of such procedures: PAC-Bayesian or PAC-Bayes bounds.Since the original PAC-Bayes bounds [163,124], these tools have been considerably improved in many directions (we will for example describe a simplified version of the localization technique of [39,41] that was missed by the community, and later rediscovered as "mutual information bounds"). Very recently, PAC-Bayes bounds received a considerable attention: for example there was workshop on PAC-Bayes at NIPS 2017, (Almost) 50 Shades of Bayesian Learning: PAC-Bayesian trends and insights, organized by B. Guedj, F. Bach and P. Germain. One of the reasons of this recent success is the successful application of these bounds to neural networks [65].An elementary introduction to PAC-Bayes theory is still missing. This is an attempt to provide such an introduction. This is a preliminary version. If you find any typo/mistake, if you think your work is not cited and should be, please let me know, and I will update the tutorial accordingly. Since 1st version: fixed (minor) problems in Theorem 4.5, in Lemma 4.6 and in Subsection 6.5.2, fixed many typos (including some in the proof of Theorem 4.3), included ref. [26,131,114].1 I don't want to scare the reader with measurability conditions, as I will not check them in this tutorial anyway. Here, the exact condition to ensure that what follows is well defined is that for any A ∈ T , the function ((x 1 , y 1 ), . . . , (x n , y n )) → [ρ((x 1 , y 1 ), . . . , (x n , y n ))] (A) is measurable. That is, ρ is a regular conditional probability.2 See the title of van Erven's tutorial [175]: "PAC-Bayes mini-tutorial: a continuous union bound". Note, however, that it is argued by Catoni in [41] that PAC-Bayes bounds are actually more than that, we will come back to this in Section 4.

show abstract

“…More theoretical studies on variational inference (using PAC-Bayes, or not) appeared at the same time or since: [50,93,122,48,145,51,20,19,138,69].…”

Section: Variational Approximationsmentioning

confidence: 99%

User-friendly introduction to PAC-Bayes bounds

Alquier¹

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Theoretical results around variational inference have mostly centered around its statistical properties, including asymptotic properties (Alquier and Ridgway, 2017;Alquier et al, 2016;Banerjee et al, 2021;Bhattacharya et al, 2020;Bhattacharya and Maiti, 2021;Bickel et al, 2013;Campbell and Li, 2019;Celisse et al, 2012;Chen and Ryzhov, 2020;Chérief-Abdellatif, 2019, 2020Chérief-Abdellatif et al, 2018;Guha et al, 2020;Hajargasht, 2019;Hall et al, 2011a,b;Han and Yang, 2019;Jaiswal et al, 2020;Knoblauch, 2019;Pati et al, 2017;Wang and Titterington, 2004;Wang et al, 2006;Blei, 2019, 2018;Westling and McCormick, 2015;Womack et al, 2013;Yang et al, 2017;You et al, 2014;Zhang and Gao, 2017), nite sample approximation error (Chen et al, 2017;Giordano et al, 2017;Huggins et al, 2020Huggins et al, , 2018Sheth and Khardon, 2017), robustness to model misspeci cation (Alquier and Ridgway, 2017;Chérief-Abdellatif et al, 2018;Medina et al, 2021;Wang and Blei, 2019), and properties in high-dimensional settings (Mukherjee and Sen, 2021;Ray and Szabó, 2021;Ray et al, 2020;…”

Section: Main Ideasmentioning

confidence: 99%

Statistical and Computational Trade-offs in Variational Inference: A Case Study in Inferential Model Selection

Bhatia¹,

Kuang²,

Ma³

et al. 2022

Preprint

View full text Add to dashboard Cite

Variational inference has recently emerged as a popular alternative to the classical Markov chain Monte Carlo (MCMC) in large-scale Bayesian inference. The core idea of variational inference is to trade statistical accuracy for computational e ciency. It aims to approximate the posterior, reducing computation costs but potentially compromising its statistical accuracy. In this work, we study this statistical and computational trade-o in variational inference via a case study in inferential model selection. Focusing on Gaussian inferential models (also known as variational approximating families) with diagonal plus low-rank precision matrices, we initiate a theoretical study of the trade-o s in two aspects, Bayesian posterior inference error and frequentist uncertainty quanti cation error. From the Bayesian posterior inference perspective, we characterize the error of the variational posterior relative to the exact posterior. We prove that, given a xed computation budget, a lower-rank inferential model produces variational posteriors with a higher statistical approximation error, but a lower computational error; it reduces variances in stochastic optimization and, in turn, accelerates convergence. From the frequentist uncertainty quanti cation perspective, we consider the precision matrix of the variational posterior as an uncertainty estimate. We nd that, relative to the true asymptotic precision, the variational approximation su ers from an additional statistical error originating from the sampling uncertainty of the data. Moreover, this statistical error becomes the dominant factor as the computation budget increases. As a consequence, for small datasets, the inferential model need not be full-rank to achieve optimal estimation error (even with unlimited computation budget). We nally demonstrate these statistical and computational trade-o s inference across empirical studies, corroborating the theoretical ndings.

show abstract

On the Robustness to Misspecification of $α$-Posteriors and Their Variational Approximations

Cited by 2 publications

References 16 publications

User-friendly introduction to PAC-Bayes bounds

User-friendly introduction to PAC-Bayes bounds

Statistical and Computational Trade-offs in Variational Inference: A Case Study in Inferential Model Selection

Contact Info

Product

Resources

About