Compact approximations to Bayesian predictive distributions

Snelson, Edward; Ghahramani, Zoubin

doi:10.1145/1102351.1102457

Cited by 43 publications

(59 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Although k-means is known to have some limitations, in our experience it usually performs reasonably well. An alternative approach would be to minimize the locations of the projected parameters {θ c ⊥ } C c=1 jointly using for example the method of Snelson and Ghahramani (2005), but this is computationally much more expensive.…”

Section: Clusteredmentioning

confidence: 99%

Projective inference in high-dimensional problems: Prediction and feature selection

Piironen¹,

Paasiniemi²,

Vehtari³

2020

Electron. J. Statist.

125

View full text Add to dashboard Cite

This paper reviews predictive inference and feature selection for generalized linear models with scarce but high-dimensional data. We demonstrate that in many cases one can benefit from a decision theoretically justified two-stage approach: first, construct a possibly non-sparse model that predicts well, and then find a minimal subset of features that characterize the predictions. The model built in the first step is referred to as the reference model and the operation during the latter step as predictive projection. The key characteristic of this approach is that it finds an excellent tradeoff between sparsity and predictive accuracy, and the gain comes from utilizing all available information including prior and that coming from the left out features. We review several methods that follow this principle and provide novel methodological contributions. We present a new projection technique that unifies two existing techniques and is both accurate and fast to compute. We also propose a way of evaluating the feature selection process using fast leave-one-out cross-validation that allows for easy and intuitive model size selection. Furthermore, we prove a theorem that helps to understand the conditions under which the projective approach could be beneficial. The key ideas are illustrated via several experiments using simulated and real world data.

show abstract

Section: Clusteredmentioning

confidence: 99%

Projective inference in high-dimensional problems: Prediction and feature selection

Piironen¹,

Paasiniemi²,

Vehtari³

2020

Electron. J. Statist.

125

View full text Add to dashboard Cite

show abstract

“…Actually, this BMA distribution minimizes on average the distance, (measured as the KL divergence, see Ref. [6]) with the true distribution [3]. The BMA distribution is better than any single model prediction [3].…”

Section: Bma Principlementioning

confidence: 99%

“…[6] we choose to seek the PEA approximation that is as close as possible to this optimal BMA classifier with respect to KL divergence. 1 Looking for an approximation of the BMA distribution is indeed a good strategy.…”

Section: Learning Strategy: Approximating Bma For Patch Functionsmentioning

confidence: 99%

“…data and task dependent) prior distribution on parameters that favours simple classifiers [5]. Other similar approximation schemes have been proposed in the past such as Kullback-Leibler (KL) projection [6] but these have been used up to now for simple classifiers only. Yet, although these methods are promising alternatives to the exact BMA approach, they have not been shown to be well adapted in practice to the automatic learning of a complex classifier.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning iteratively a classifier with the Bayesian Model Averaging Principle

Sicard

Artières

Petit

2008

Pattern Recognition

View full text Add to dashboard Cite

“…Our work brings together ideas often referred to in the literature as knowledge distillation (Hinton et al, 2015), model compression (Bucilȃ et al, 2006), compact approximations (Snelson and Ghahramani, 2005a), mimicking models (Ba and Caruana, 2014), and teacher-student models (Romero et al, 2015). As a technical term, "knowledge distillation" was introduced only recently by Hinton et al (2015), who used it to mean model compression.…”

Section: Introductionmentioning

confidence: 99%

Distilling Model Knowledge

Papamakarios

2015

Preprint

View full text Add to dashboard Cite

Top-performing machine learning systems, such as deep neural networks, large ensembles and complex probabilistic graphical models, can be expensive to store, slow to evaluate and hard to integrate into larger systems. Ideally, we would like to replace such cumbersome models with simpler models that perform equally well.In this thesis, we study knowledge distillation, the idea of extracting the knowledge contained in a complex model and injecting it into a more convenient model. We present a general framework for knowledge distillation, whereby a convenient model of our choosing learns how to mimic a complex model, by observing the latter's behaviour and being penalized whenever it fails to reproduce it.We develop our framework within the context of three distinct machine learning applications: (a) model compression, where we compress large discriminative models, such as ensembles of neural networks, into models of much smaller size; (b) compact predictive distributions for Bayesian inference, where we distil large bags of MCMC samples into compact predictive distributions in closed form; (c) intractable generative models, where we distil unnormalizable models such as RBMs into tractable models such as NADEs.We contribute to the state of the art with novel techniques and ideas. In model compression, we describe and implement derivative matching, which allows for better distillation when data is scarce. In compact predictive distributions, we introduce online distillation, which allows for significant savings in memory. Finally, in intractable generative models, we show how to use distilled models to robustly estimate intractable quantities of the original model, such as its intractable partition function. myself fortunate to have been in the privileged group of individuals that have been supervised by Iain.I am grateful to all fellow PhD students in the Institute for Adaptive and Neural Computation and the Centre for Doctoral Training in Data Science with whom I have had numerous discussions about machine learning (and beyond!). Special thanks go to Harri Edwards and Krzysztof Geras for regularly pointing me to good papers.None of this would have existed had it not been for the directors of the Centre for Doctoral Training in Data Science, Prof. Chris Williams and Dr Charles Sutton. Many thanks to them for our regular interactions and making everything possible.

show abstract

Compact approximations to Bayesian predictive distributions

Cited by 43 publications

References 6 publications

Projective inference in high-dimensional problems: Prediction and feature selection

Projective inference in high-dimensional problems: Prediction and feature selection

Learning iteratively a classifier with the Bayesian Model Averaging Principle

Distilling Model Knowledge

Contact Info

Product

Resources

About