Speeding Up MCMC by Efficient Data Subsampling

Quiroz, Matias; Kohn, Robert; Villani, Mattias; Tran, Minh‐Ngoc

doi:10.1080/01621459.2018.1448827

Cited by 122 publications

(167 citation statements)

References 39 publications

Supporting

Mentioning

167

Contrasting

Order By: Relevance

“…Typically a small fraction of observations are included, hence speeding up the execution time significantly compared to MH. However, the augmentation scheme severely affects the mixing of the Firefly algorithm and it has been demonstrated to perform poorly compared to MH and other subsampling approaches (Bardenet et al, 2015;Quiroz et al, 2016Quiroz et al, , 2017. We conclude that delayed acceptance, out of the discussed methods, seems to be the only feasible route to obtain exact simulation via data subsampling.…”

Section: Introductionmentioning

confidence: 95%

“…Other authors use a subsample of the data in each MCMC iteration to speed up the algorithm, see e.g. Korattikara et al (2014), Bardenet et al (2014), Maclaurin and Adams (2014), Maire et al (2015), Bardenet et al (2015) and Quiroz et al (2016Quiroz et al ( , 2017. Finally, delayed acceptance MCMC has been used to speed up computations (Banterle et al, 2014;Payne and Mallick, 2015).…”

Section: Introductionmentioning

confidence: 99%

“…This typically requires computations using the full data set, naturally defeating the purpose of subsampling. Quiroz et al (2017) instead suggests to compute a soft lower bound which is defined to be a lower bound with a high probability. Since the estimate can occasionally be negative, the pseudo-marginal in Quiroz et al (2017) instead targets an absolute measure following Lyne et al (2015), who show that the draws can be corrected with an importance sampling type of step to estimate expectations of posterior functions exactly.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Speeding up MCMC by Delayed Acceptance and Data Subsampling

Quiroz

Tran

Villani

et al. 2017

Journal of Computational and Graphical Statistics

Self Cite

View full text Add to dashboard Cite

Abstract. The complexity of the Metropolis-Hastings (MH) algorithm arises from the requirement of a likelihood evaluation for the full data set in each iteration. Payne and Mallick (2015) propose to speed up the algorithm by a delayed acceptance approach where the acceptance decision proceeds in two stages. In the first stage, an estimate of the likelihood based on a random subsample determines if it is likely that the draw will be accepted and, if so, the second stage uses the full data likelihood to decide upon final acceptance. Evaluating the full data likelihood is thus avoided for draws that are unlikely to be accepted.We propose a more precise likelihood estimator which incorporates auxiliary information about the full data likelihood while only operating on a sparse set of the data. We prove that the resulting delayed acceptance MH is more efficient compared to that of Payne and Mallick (2015). The caveat of this approach is that the full data set needs to be evaluated in the second stage. We therefore propose to substitute this evaluation by an estimate and construct a state-dependent approximation thereof to use in the first stage. This results in an algorithm that (i) can use a smaller subsample m by leveraging on recent advances in Pseudo-Marginal MH (PMMH) and (ii) is provably within O(m −2 ) of the true posterior.

show abstract

Section: Introductionmentioning

confidence: 95%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Speeding up MCMC by Delayed Acceptance and Data Subsampling

Quiroz

Tran

Villani

et al. 2017

Journal of Computational and Graphical Statistics

Self Cite

View full text Add to dashboard Cite

show abstract

“…The log-likelihood is estimated by an unbiased subsampled version, and an unbiased estimator of the likelihood is obtained by correcting the bias of the exponentiation of this estimator. Quiroz, Villani and Kohn (2014) proposed subsampling the data using probability proportional-to-size (PPS) sampling to obtain an approximately unbiased estimate of the likelihood which is used in the MH acceptance step. The subsampling approach was further improved in Quiroz, Villani and Kohn (2015) using the efficient and robust difference estimator from the survey sampling literature.…”

Section: Methodsmentioning

confidence: 99%

Statistical methods and computing for big data

Wang

Chen

Schifano

et al. 2016

Statistics and Its Interface

107

View full text Add to dashboard Cite

Big data are data on a massive scale in terms of volume, intensity, and complexity that exceed the capacity of standard analytic tools. They present opportunities as well as challenges to statisticians. The role of computational statisticians in scientific discovery from big data analyses has been under-recognized even by peer statisticians. This article summarizes recent methodological and software developments in statistics that address the big data challenges. Methodologies are grouped into three classes: subsampling-based, divide and conquer, and online updating for stream data. As a new contribution, the online updating approach is extended to variable selection with commonly used criteria, and their performances are assessed in a simulation study with stream data. Software packages are summarized with focuses on the open source and packages, covering recent tools that help break the barriers of computer memory and computing power. Some of the tools are illustrated in a case study with a logistic regression for the chance of airline delay.

show abstract

“…Recently, there have been several articles working with a random subset of the full dataset at a time as a way to speed up computations (Maclaurin and Adams, 2014;Quiroz et al, 2014). These methods require access to the full dataset, that needs to remain on-hold, and therefore require fre-3…”

Section: Accepted Manuscriptmentioning

confidence: 99%

Parallel Variational Bayes for Large Datasets With an Application to Generalized Linear Mixed Models

Tran¹,

Nott²,

Kuk³

et al. 2016

Journal of Computational and Graphical Statistics

Self Cite

View full text Add to dashboard Cite

The article develops a hybrid Variational Bayes algorithm that combines the mean-field and stochastic linear regression fixed-form Variational Bayes methods. The new estimation algorithm can be used to approximate any posterior without relying on conjugate priors. We propose a divide and recombine strategy for the analysis of large datasets, which partitions a large dataset into smaller subsets and then combines the variational distributions that have been learnt in parallel on each separate subset using the hybrid Variational Bayes algorithm. We also describe an efficient model selection strategy using cross validation, which is straightforward to implement as a by-product of the parallel run. The proposed method is applied to fitting generalized linear mixed models. The computational efficiency of the parallel and hybrid Variational Bayes algorithm is demonstrated on several simulated and real datasets. Supplementary material for this article is available online.

show abstract

Speeding Up MCMC by Efficient Data Subsampling

Cited by 122 publications

References 39 publications

Speeding up MCMC by Delayed Acceptance and Data Subsampling

Speeding up MCMC by Delayed Acceptance and Data Subsampling

Statistical methods and computing for big data

Parallel Variational Bayes for Large Datasets With an Application to Generalized Linear Mixed Models

Contact Info

Product

Resources

About