Asymptotic consistency of loss‐calibrated variational Bayes

Jaiswal, Prateek; Honnappa, Harsha; Rao, Vinayak

doi:10.1002/sta4.258

Cited by 3 publications

(3 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, leveraging these to establish the PAC-Bayes bounds for the KL-VB posterior is a challenging effort that we leave to future papers. Finally it is of interest to generalize our PAC-bounds to posterior approximations beyond KL-variational inference, such as

-Rényi posterior approximations [ 6 ], and loss-calibrated posterior approximations [ 24 , 25 ].…”

Section: Discussionmentioning

confidence: 99%

PAC-Bayes Bounds on Variational Tempered Posteriors for Markov Models

Banerjee

Rao

Honnappa

2021

Entropy

Self Cite

View full text Add to dashboard Cite

Datasets displaying temporal dependencies abound in science and engineering applications, with Markov models representing a simplified and popular view of the temporal dependence structure. In this paper, we consider Bayesian settings that place prior distributions over the parameters of the transition kernel of a Markov model, and seek to characterize the resulting, typically intractable, posterior distributions. We present a Probably Approximately Correct (PAC)-Bayesian analysis of variational Bayes (VB) approximations to tempered Bayesian posterior distributions, bounding the model risk of the VB approximations. Tempered posteriors are known to be robust to model misspecification, and their variational approximations do not suffer the usual problems of over confident approximations. Our results tie the risk bounds to the mixing and ergodic properties of the Markov data generating model. We illustrate the PAC-Bayes bounds through a number of example Markov models, and also consider the situation where the Markov model is misspecified.

show abstract

-Rényi posterior approximations [ 6 ], and loss-calibrated posterior approximations [ 24 , 25 ].…”

Section: Discussionmentioning

confidence: 99%

PAC-Bayes Bounds on Variational Tempered Posteriors for Markov Models

Banerjee

Rao

Honnappa

2021

Entropy

Self Cite

View full text Add to dashboard Cite

show abstract

Section: Discussionmentioning

confidence: 99%

PAC-Bayes Bounds on Variational Tempered Posteriors for Markov Models

Banerjee¹,

Rao²,

Honnappa³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Datasets displaying temporal dependencies abound in science and engineering applications, with Markov models representing a simplified and popular view of the temporal dependence structure. In this paper, we consider Bayesian settings that place prior distributions over the parameters of the transition kernel of a Markov model, and seeks to characterize the resulting, typically intractable, posterior distributions. We present a PAC-Bayesian analysis of variational Bayes (VB) approximations to tempered Bayesian posterior distributions, bounding the model risk of the VB approximations. Tempered posteriors are known to be robust to model misspecification, and their variational approximations do not suffer the usual problems of over confident approximations. Our results tie the risk bounds to the mixing and ergodic properties of the Markov data generating model. We illustrate the PAC-Bayes bounds through a number of example Markov models, and also consider the situation where the Markov model is misspecified.

show abstract

“…In the online setting, online variational approximations are studied by [31,32] and led to the first scaling of Bayesian principles to state-of-the-art neural networks [38]. In the i.i.d setting, a series of paper established the first theoretical results on variational inference, for some of them through a connection with PAC-Bayes bounds [5,43,20,16,14,47,4,48,46,29,49,15]. Up to our knowledge, the only regret bound for online variational inference can be found in [17].…”

Section: Related Workmentioning

confidence: 99%

Non-exponentially weighted aggregation: regret bounds for unbounded loss functions

Alquier¹

2020

Preprint

View full text Add to dashboard Cite

We tackle the problem of online optimization with a general, possibly unbounded, loss function. It is well known that the exponentially weighted aggregation strategy (EWA) leads to a regret in √ T after T steps, under the assumption that the loss is bounded. The online gradient algorithm (OGA) has a regret in √ T when the loss is convex and Lipschitz. In this paper, we study a generalized aggregation strategy, where the weights do no longer necessarily depend exponentially on the losses. Our strategy can be interpreted as the minimization of the expected losses plus a penalty term. When the penalty term is the Kullback-Leibler divergence, we obtain EWA as a special case, but using alternative divergences lead to a regret bounds for unbounded, not necessarily convex losses. However, the cost is a worst regret bound in some cases.

show abstract

Asymptotic consistency of loss‐calibrated variational Bayes

Cited by 3 publications

References 16 publications

PAC-Bayes Bounds on Variational Tempered Posteriors for Markov Models

PAC-Bayes Bounds on Variational Tempered Posteriors for Markov Models

PAC-Bayes Bounds on Variational Tempered Posteriors for Markov Models

Non-exponentially weighted aggregation: regret bounds for unbounded loss functions

Contact Info

Product

Resources

About