Subsampling MCMC - an Introduction for the Survey Statistician

Quiroz, Matias; Villani, Mattias; Kohn, Robert; Tran, Minh‐Ngoc; Dang, Khue‐Dung

doi:10.1007/s13171-018-0153-7

Cited by 15 publications

(10 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It would, thus, be interesting to extend the subsampling ideas put forth in this article to that context by combining them with INLA. Another interesting idea for future research is to apply the proposed methods of Quiroz et al (2018) when doing the final calculation of the deviance. This would further improve the speed of the only step in the current algorithm that depends on the full set of data.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

A subsampling approach for Bayesian model selection

Lachmann¹,

Storvik²,

Frommlet³

et al. 2022

Preprint

View full text Add to dashboard Cite

It is common practice to use Laplace approximations to compute marginal likelihoods in Bayesian versions of generalised linear models (GLM). Marginal likelihoods combined with model priors are then used in different search algorithms to compute the posterior marginal probabilities of models and individual covariates. This allows performing Bayesian model selection and model averaging. For large sample sizes, even the Laplace approximation becomes computationally challenging because the optimisation routine involved needs to evaluate the likelihood on the full set of data in multiple iterations. As a consequence, the algorithm is not scalable for large datasets. To address this problem, we suggest using a version of a popular batch stochastic gradient descent (BSGD) algorithm for estimating the marginal likelihood of a GLM by subsampling from the data. We further combine the algorithm with Markov chain Monte Carlo (MCMC) based methods for Bayesian model selection and provide some theoretical results on convergence of the estimates. Finally, we report results from experiments illustrating the performance of the proposed algorithm.

show abstract

Section: Discussionmentioning

confidence: 99%

“…This would further improve the speed of the only step in the current algorithm that depends on the full set of data. Yet, the methods from Quiroz et al (2018) would break the regularity conditions of our Theorems 1 and 2, which means that additional theoretical studies would also be required to be able to incorporate it into our framework.…”

Section: Discussionmentioning

confidence: 99%

A subsampling approach for Bayesian model selection

Lachmann¹,

Storvik²,

Frommlet³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…As a result, the samples generated with this algorithm have a smaller size than the initial dataset X n . As the bootstrap algorithm, this one has been widely studied and used in the most different fields, from genomics [51,52] to survey science [53,54], finance [55,56] and, of course, statistics [26,57,58]. The two reviews [26,59] can be consulted by the interested reader.…”

Section: Subsamplingmentioning

confidence: 99%

Estimating Neural Network’s Performance with Bootstrap: A Tutorial

Michelucci

Venturini

2021

MAKE

View full text Add to dashboard Cite

Neural networks present characteristics where the results are strongly dependent on the training data, the weight initialisation, and the hyperparameters chosen. The determination of the distribution of a statistical estimator, as the Mean Squared Error (MSE) or the accuracy, is fundamental to evaluate the performance of a neural network model (NNM). For many machine learning models, as linear regression, it is possible to analytically obtain information as variance or confidence intervals on the results. Neural networks present the difficulty of not being analytically tractable due to their complexity. Therefore, it is impossible to easily estimate distributions of statistical estimators. When estimating the global performance of an NNM by estimating the MSE in a regression problem, for example, it is important to know the variance of the MSE. Bootstrap is one of the most important resampling techniques to estimate averages and variances, between other properties, of statistical estimators. In this tutorial, the application of resampling techniques (including bootstrap) to the evaluation of neural networks’ performance is explained from both a theoretical and practical point of view. The pseudo-code of the algorithms is provided to facilitate their implementation. Computational aspects, as the training time, are discussed, since resampling techniques always require simulations to be run many thousands of times and, therefore, are computationally intensive. A specific version of the bootstrap algorithm is presented that allows the estimation of the distribution of a statistical estimator when dealing with an NNM in a computationally effective way. Finally, algorithms are compared on both synthetically generated and real data to demonstrate their performance.

show abstract

“…However, learning from training data to improve inference on future data with a hierarchical model is computationally inefficient -if a new box is presented, one has to add observations of the new box to the previously available data and re-run inference on the extended data set. Inference performance can be improved by employing data subsampling [Bardenet et al 2014[Bardenet et al , 2017Korattikara et al 2014;Maclaurin and Adams 2014;Quiroz et al 2018], but the whole training data set still needs to be kept and made accessible to the inference algorithm. A hierarchical model cannot 'compress', or summarize, training data for efficient inference on future observations.…”

Section: Introductionmentioning

confidence: 99%

How To Train Your Program: a Probabilistic Programming Pattern for Bayesian Learning From Data

Tolpin¹

2021

Preprint

View full text Add to dashboard Cite

We present a Bayesian approach to machine learning with probabilistic programs. In our approach, training on available data is implemented as inference on a hierarchical model. The posterior distribution of model parameters is then used to stochastically condition a complementary model, such that inference on new data yields the same posterior distribution of latent parameters corresponding to the new data as inference on a hierachical model on the combination of both previously available and new data, at a lower computation cost. We frame the approach as a design pattern of probabilistic programming referred to herein as 'stump and fungus', and illustrate realization of the pattern on a didactic case study.

show abstract

Subsampling MCMC - an Introduction for the Survey Statistician

Cited by 15 publications

References 47 publications

A subsampling approach for Bayesian model selection

A subsampling approach for Bayesian model selection

Estimating Neural Network’s Performance with Bootstrap: A Tutorial

How To Train Your Program: a Probabilistic Programming Pattern for Bayesian Learning From Data

Contact Info

Product

Resources

About