Distribution-Dependent PAC-Bayes Priors

Lever, Guy; Laviolette, François; Shawe‐Taylor, John

doi:10.1007/978-3-642-16108-7_13

Cited by 31 publications

(26 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…√ n using Stirling's approximation of the factorial [19]. For the sake of simplicity we restrict ourselves to the slightly weaker bound (18), although all results that are based on Lemma 2 can be slightly improved by using the tighter bound.…”

Section: Appendix a Proofs Of The Results For Individual Martingalesmentioning

confidence: 99%

“…A combination of PAC-Bayesian analysis with Hoeffding-Azuma's inequality was applied by Lever et. al [18] in the analysis of U-statistics. The results presented here are both tighter and more general, and make it possible to apply 1 The complete statement of Donsker-Varadhan's variational formula for relative entropy states that under appropriate conditions KL(ρ π) = sup φ φ, ρ − ln e φ , π , where the supremum is achieved by φ(h) = ln ρ(h) π(h) .…”

Section: − −−−−−−−− → Timementioning

confidence: 99%

See 1 more Smart Citation

PAC-Bayesian Inequalities for Martingales

Seldin

Laviolette

Cesa-Bianchi

et al. 2012

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

Abstract-We present a set of high-probability inequalities that control the concentration of weighted averages of multiple (possibly uncountably many) simultaneously evolving and interdependent martingales. Our results extend the PAC-Bayesian (probably approximately correct) analysis in learning theory from the i.i.d. setting to martingales opening the way for its application to importance weighted sampling, reinforcement learning, and other interactive learning domains, as well as many other domains in probability theory and statistics, where martingales are encountered. We also present a comparison inequality that bounds the expectation of a convex function of a martingale difference sequence shifted to the interval by the expectation of the same function of independent Bernoulli random variables. This inequality is applied to derive a tighter analog of Hoeffding-Azuma's inequality.

show abstract

Section: Appendix a Proofs Of The Results For Individual Martingalesmentioning

confidence: 99%

Section: − −−−−−−−− → Timementioning

confidence: 99%

PAC-Bayesian Inequalities for Martingales

Seldin

Laviolette

Cesa-Bianchi

et al. 2012

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

show abstract

“…To do so, we may need to fix an "informed prior" to minimise the KL divergence with an interesting posterior. This idea has been studied by [16,17] and, more recently, by Mhammedi et al [18], Rivasplata et al [5], among others. We will adapt it to our problem in the simplest way.…”

Section: Lemma 1 For Any Given Kmentioning

confidence: 99%

PAC-Bayes Unleashed: Generalisation Bounds with Unbounded Losses

Maxime

Guedj

Rivasplata

et al. 2021

Entropy

Self Cite

View full text Add to dashboard Cite

We present new PAC-Bayesian generalisation bounds for learning problems with unbounded loss functions. This extends the relevance and applicability of the PAC-Bayes learning framework, where most of the existing literature focuses on supervised learning problems with a bounded loss function (typically assumed to take values in the interval [0;1]). In order to relax this classical assumption, we propose to allow the range of the loss to depend on each predictor. This relaxation is captured by our new notion of HYPothesis-dependent rangE (HYPE). Based on this, we derive a novel PAC-Bayesian generalisation bound for unbounded loss functions, and we instantiate it on a linear regression problem. To make our theory usable by the largest audience possible, we include discussions on actual computation, practicality and limitations of our assumptions.

show abstract

“…In recent years, it has been widely applied to regression [7], density estimation [9], and analysis for non-Ll.D. data [10][11].…”

Section: Valiant Proposed the Probably Approximately Correct (Pac)mentioning

confidence: 95%

Optimization of MCMC sampling algorithm for the calculation of PAC-Bayes bound

Tang

Zhao

Gong

et al. 2013

2013 International Conference on Machine Learning and Cybernetics

View full text Add to dashboard Cite

PAC-Bayes bound provides a formal framework for deducing the tightest risk bounds of the classifiers. After formulating the concept space as a Reproducing Kernel Hilbert Space (RKHS), the Markov Chain Monte Carlo (MCMC) sampling algorithm for simulating posterior distributions of the concept space can realize the calculation of PAC-Bayes bound. A major issue is the computational complexity in geometric growth when the dimension of concept space increases. In this paper, we store a portion of the sampling data and calculate its variance, after which the variance minimization method is proposed to investigate the support vectors. Finally, we optimize the support vectors coupled with their weight vectors, and compare the PAC-Bayes bounds. The experimental results of our artificial data sets in low-dimensional spaces show that the optimization is reasonable and effective in practice.

show abstract

Distribution-Dependent PAC-Bayes Priors

Cited by 31 publications

References 12 publications

PAC-Bayesian Inequalities for Martingales

PAC-Bayesian Inequalities for Martingales

PAC-Bayes Unleashed: Generalisation Bounds with Unbounded Losses

Optimization of MCMC sampling algorithm for the calculation of PAC-Bayes bound

Contact Info

Product

Resources

About