A PAC-Bayes Bound for Tailored Density Estimation

Higgs, Matthew; Shawe‐Taylor, John

doi:10.1007/978-3-642-16108-7_15

Cited by 10 publications

(7 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…PAC-Bayes bounds were initially used for classification [8]. In recent years, it has been widely applied to regression [7], density estimation [9], and analysis for non-Ll.D. data [10][11].…”

Section: Valiant Proposed the Probably Approximately Correct (Pac)mentioning

confidence: 99%

“…According to the probability density of prior and posterior distribution, KL values can be calculated. From the equation (3), KL value is defined by KL(Q I P) = sum(ln(Q I P)) / len(Q) (9) The average true error QD can be calculated using a binary search approach in our previous research [12]. The pseudo code for the calculation of KL values and average true error is listed in algorithm (2).…”

Section: Calculation Of Kl Values and P Ac-bayes Boundmentioning

confidence: 99%

See 1 more Smart Citation

Optimization of MCMC sampling algorithm for the calculation of PAC-Bayes bound

Tang

Zhao

Gong

et al. 2013

2013 International Conference on Machine Learning and Cybernetics

View full text Add to dashboard Cite

PAC-Bayes bound provides a formal framework for deducing the tightest risk bounds of the classifiers. After formulating the concept space as a Reproducing Kernel Hilbert Space (RKHS), the Markov Chain Monte Carlo (MCMC) sampling algorithm for simulating posterior distributions of the concept space can realize the calculation of PAC-Bayes bound. A major issue is the computational complexity in geometric growth when the dimension of concept space increases. In this paper, we store a portion of the sampling data and calculate its variance, after which the variance minimization method is proposed to investigate the support vectors. Finally, we optimize the support vectors coupled with their weight vectors, and compare the PAC-Bayes bounds. The experimental results of our artificial data sets in low-dimensional spaces show that the optimization is reasonable and effective in practice.

show abstract

Section: Valiant Proposed the Probably Approximately Correct (Pac)mentioning

confidence: 99%

Section: Calculation Of Kl Values and P Ac-bayes Boundmentioning

confidence: 99%

Optimization of MCMC sampling algorithm for the calculation of PAC-Bayes bound

Tang

Zhao

Gong

et al. 2013

2013 International Conference on Machine Learning and Cybernetics

View full text Add to dashboard Cite

show abstract

“…Different from early bounds that often rely on the complexity measures of the considered function classes, the recent PAC-Bayes bounds (McAllester, 1999;Seeger, 2002;Langford, 2005) give the tightest predictions of the generalization performance, for which the prior and posterior distributions of learners are involved on top of the PAC (Probably Approximately Correct) learning setting (Catoni, 2007;Germain et al, 2009). Beyond the common supervised learning, PAC-Bayes analysis has also been applied to other tasks, e.g., density estimation (Seldin and Tishby, 2010;Higgs and Shawe-Taylor, 2010) and reinforcement learning (Seldin et al, 2012).…”

Section: Introductionmentioning

confidence: 99%

PAC-Bayes analysis of multi-view learning

2017

Self Cite

View full text Add to dashboard Cite

This paper presents eight PAC-Bayes bounds to analyze the generalization performance of multi-view classifiers. These bounds adopt data dependent Gaussian priors which emphasize classifiers with high view agreements. The center of the prior for the first two bounds is the origin, while the center of the prior for the third and fourth bounds is given by a data dependent vector. An important technique to obtain these bounds is two derived logarithmic determinant inequalities whose difference lies in whether the dimensionality of data is involved. The centers of the fifth and sixth bounds are calculated on a separate subset of the training set. The last two bounds use unlabeled data to represent view agreements and are thus applicable to semi-supervised multi-view learning. We evaluate all the presented multi-view PAC-Bayes bounds on benchmark data and compare them with previous single-view PAC-Bayes bounds. The usefulness and performance of the multi-view bounds are discussed.

show abstract

“…data for a long time. The issue of treating non-independent samples was partially addressed only recently by Ralaivola et al (2010) and Lever et al (2010) (their approaches are also suitable for density estimation (Higgs and Shawe-Taylor, 2010). The solution of Ralaivola et al (2010) essentially boils down to breaking the sample into independent (or almost independent) subsets (which also reduces the effective sample size to the number of independent subsets).…”

Section: Introductionmentioning

confidence: 99%

PAC-Bayesian Analysis of Martingales and Multiarmed Bandits

Seldin,

Laviolette,

Shawe-Taylor

et al. 2011

Preprint

Self Cite

View full text Add to dashboard Cite

We present two alternative ways to apply PAC-Bayesian analysis to sequences of dependent random variables. The first is based on a new lemma that enables to bound expectations of convex functions of certain dependent random variables by expectations of the same functions of independent Bernoulli random variables. This lemma provides an alternative tool to Hoeffding-Azuma inequality to bound concentration of martingale values. Our second approach is based on integration of Hoeffding-Azuma inequality with PAC-Bayesian analysis. We also introduce a way to apply PAC-Bayesian analysis in situation of limited feedback. We combine the new tools to derive PAC-Bayesian generalization and regret bounds for the multiarmed bandit problem. Although our regret bound is not yet as tight as state-of-the-art regret bounds based on other well-established techniques, our results significantly expand the range of potential applications of PAC-Bayesian analysis and introduce a new analysis tool to reinforcement learning and many other fields, where martingales and limited feedback are encountered.

show abstract

A PAC-Bayes Bound for Tailored Density Estimation

Cited by 10 publications

References 13 publications

Optimization of MCMC sampling algorithm for the calculation of PAC-Bayes bound

Optimization of MCMC sampling algorithm for the calculation of PAC-Bayes bound

PAC-Bayes analysis of multi-view learning

PAC-Bayesian Analysis of Martingales and Multiarmed Bandits

Contact Info

Product

Resources

About