Meta-Strategy for Learning Tuning Parameters with Guarantees

Meunier, Dimitri; Alquier, Pierre

doi:10.3390/e23101257

Cited by 3 publications

(1 citation statement)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This improvement differs from one application to the other, for example: learn a better parameter space Θ T +1 ⊂ Θ, learn a better prior, learn better hyperparameters like λ... PAC-Bayes bounds for meta-learning were studied in[143,12,96,154,112,129,113]. I believe PAC-Bayes bound are particularly convenient for meta-learning problems, and thus that this direction of research is very promising.6 Related approaches in statistics and machine learning theoryIn this section, we list some well-known connections between PAC-Bayes theory and other approaches in statistics and machine learning.…”

mentioning

confidence: 99%

User-friendly introduction to PAC-Bayes bounds

Alquier¹

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Aggregated predictors are obtained by making a set of basic predictors vote according to some weights, that is, to some probability distribution.Randomized predictors are obtained by sampling in a set of basic predictors, according to some prescribed probability distribution.Thus, aggregated and randomized predictors have in common that they are not defined by a minimization problem, but by a probability distribution on the set of predictors. In statistical learning theory, there is a set of tools designed to understand the generalization ability of such procedures: PAC-Bayesian or PAC-Bayes bounds.Since the original PAC-Bayes bounds [163,124], these tools have been considerably improved in many directions (we will for example describe a simplified version of the localization technique of [39,41] that was missed by the community, and later rediscovered as "mutual information bounds"). Very recently, PAC-Bayes bounds received a considerable attention: for example there was workshop on PAC-Bayes at NIPS 2017, (Almost) 50 Shades of Bayesian Learning: PAC-Bayesian trends and insights, organized by B. Guedj, F. Bach and P. Germain. One of the reasons of this recent success is the successful application of these bounds to neural networks [65].An elementary introduction to PAC-Bayes theory is still missing. This is an attempt to provide such an introduction. This is a preliminary version. If you find any typo/mistake, if you think your work is not cited and should be, please let me know, and I will update the tutorial accordingly. Since 1st version: fixed (minor) problems in Theorem 4.5, in Lemma 4.6 and in Subsection 6.5.2, fixed many typos (including some in the proof of Theorem 4.3), included ref. [26,131,114].1 I don't want to scare the reader with measurability conditions, as I will not check them in this tutorial anyway. Here, the exact condition to ensure that what follows is well defined is that for any A ∈ T , the function ((x 1 , y 1 ), . . . , (x n , y n )) → [ρ((x 1 , y 1 ), . . . , (x n , y n ))] (A) is measurable. That is, ρ is a regular conditional probability.2 See the title of van Erven's tutorial [175]: "PAC-Bayes mini-tutorial: a continuous union bound". Note, however, that it is argued by Catoni in [41] that PAC-Bayes bounds are actually more than that, we will come back to this in Section 4.

show abstract

mentioning

confidence: 99%

User-friendly introduction to PAC-Bayes bounds

Alquier¹

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

A reduced-rank approach to predicting multiple binary responses through machine learning

Mai

2023

Stat Comput

View full text Add to dashboard Cite

This paper investigates the problem of simultaneously predicting multiple binary responses by utilizing a shared set of covariates. Our approach incorporates machine learning techniques for binary classification, without making assumptions about the underlying observations. Instead, our focus lies on a group of predictors, aiming to identify the one that minimizes prediction error. Unlike previous studies that primarily address estimation error, we directly analyze the prediction error of our method using PAC-Bayesian bounds techniques. In this paper, we introduce a pseudo-Bayesian approach capable of handling incomplete response data. Our strategy is efficiently implemented using the Langevin Monte Carlo method. Through simulation studies and a practical application using real data, we demonstrate the effectiveness of our proposed method, producing comparable or sometimes superior results compared to the current state-of-the-art method.

show abstract