Implicitly adaptive importance sampling

Paananen, Topi; Piironen, Juho; Bürkner, Paul - Christian; Vehtari, Aki

doi:10.1007/s11222-020-09982-2

Cited by 34 publications

(25 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More generally, our work support recent evidence, both in the Bayesian and frequentist literature [Beirami et al, 2017, Rad et al, 2020, Giordano et al, 2019, Paananen et al, 2021, that LOO-CV criteria can be reliably approximated with a computational cost comparable to the one of a single model fit, thus being not only statistically appealing but also computationally practical. In this sense LOO-CV can be computationally cheaper than k-fold CV by a factor of k, since the latter requires fitting k separate models and is not easily amenable to the same importance sampling tricks as LOO-CV.…”

Section: Introductionsupporting

confidence: 89%

“…A notable example that we compare with in simulations later on is the Pareto-smoothed importance sampling methodology of [Vehtari et al, 2017] implemented in the popular loo R package [Vehtari et al, 2020]. See also Alqallaf and Gustafson [2001], Bornn et al [2010], Rischard et al [2018], Paananen et al [2021] for other work in the area, and Section 4.3.1 for comparison with some of those.…”

Section: θ|Y)mentioning

confidence: 99%

See 1 more Smart Citation

Robust leave-one-out cross-validation for high-dimensional Bayesian models

Silva¹,

Zanella²

2022

Preprint

View full text Add to dashboard Cite

Leave-one-out cross-validation (LOO-CV) is a popular method for estimating outof-sample predictive accuracy. However, computing LOO-CV criteria can be computationally expensive due to the need to fit the model multiple times. In the Bayesian context, importance sampling provides a possible solution but classical approaches can easily produce estimators whose variance is infinite, making them potentially unreliable. Here we propose and analyze a novel mixture estimator to compute Bayesian LOO-CV criteria. Our method retains the simplicity and computational convenience of classical approaches, while guaranteeing finite variance of the resulting estimators. Both theoretical and numerical results are provided to illustrate the improved robustness and efficiency. The computational benefits are particularly significant in highdimensional problems, allowing to perform Bayesian LOO-CV for a broader range of models. The proposed methodology is easily implementable in standard probabilistic programming software and has a computational cost roughly equivalent to fitting the original model once.

show abstract

Section: Introductionsupporting

confidence: 89%

Section: θ|Y)mentioning

confidence: 99%

Robust leave-one-out cross-validation for high-dimensional Bayesian models

Silva¹,

Zanella²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Larger values indicate that the calculated utilities u (i) k and u (i) * for such observation i have high variance and can be biased (optimistic). In such cases, better estimates can be obtained by iterative moment matching LOO (Paananen et al, 2020) or K-fold validation. In Section 7.2 we demonstrate empirically that even when a few k-values exceed this threshold, the relative utility estimate ( 22) can be nearly unbiased since the bias in both u (i) k and u (i) * tends to cancel out in the subtraction.…”

Section: Leave-one-out Cross-validationmentioning

confidence: 99%

Projective inference in high-dimensional problems: Prediction and feature selection

Piironen¹,

Paasiniemi²,

Vehtari³

2020

Electron. J. Statist.

Self Cite

125

View full text Add to dashboard Cite

This paper reviews predictive inference and feature selection for generalized linear models with scarce but high-dimensional data. We demonstrate that in many cases one can benefit from a decision theoretically justified two-stage approach: first, construct a possibly non-sparse model that predicts well, and then find a minimal subset of features that characterize the predictions. The model built in the first step is referred to as the reference model and the operation during the latter step as predictive projection. The key characteristic of this approach is that it finds an excellent tradeoff between sparsity and predictive accuracy, and the gain comes from utilizing all available information including prior and that coming from the left out features. We review several methods that follow this principle and provide novel methodological contributions. We present a new projection technique that unifies two existing techniques and is both accurate and fast to compute. We also propose a way of evaluating the feature selection process using fast leave-one-out cross-validation that allows for easy and intuitive model size selection. Furthermore, we prove a theorem that helps to understand the conditions under which the projective approach could be beneficial. The key ideas are illustrated via several experiments using simulated and real world data.

show abstract

“…To further alleviate issues with importance sampling, we use Pareto smoothed importance sampling (PSIS; Vehtari et al, 2019), which stabilises the importance weights in an efficient, self-diagnosing and trustworthy manner by modelling the upper tail of the importance weights with a generalised Pareto distribution. In cases where PSIS does not perform adequately, weights are further adapted with importance weighted moment matching (IWMM; Paananen et al, 2021), which is a generic adaptive importance sampling algorithm that improves the implicit proposal distribution by iterative weighted moment matching. The combination of using a continuous parameter to control the amount of perturbation, along with PSIS and IWMM, allows for a reliable and self-diagnosing method of estimating properties of perturbed posteriors.…”

Section: Estimating Properties Of Perturbed Posteriorsmentioning

confidence: 99%

Detecting and diagnosing prior and likelihood sensitivity with power-scaling

Kallioinen¹,

Paananen²,

Bürkner³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Determining the sensitivity of the posterior to perturbations of the prior and likelihood is an important part of the Bayesian workflow. We introduce a practical and computationally efficient sensitivity analysis approach that is applicable to a wide range of models, based on power-scaling perturbations. We suggest a diagnostic based on this that can indicate the presence of prior-data conflict or likelihood noninformativity. The approach can be easily included in Bayesian workflows with minimal work by the model builder. We present the implementation of the approach in our new R package priorsense and demonstrate the workflow on case studies of real data.

show abstract

Implicitly adaptive importance sampling

Cited by 34 publications

References 54 publications

Robust leave-one-out cross-validation for high-dimensional Bayesian models

Robust leave-one-out cross-validation for high-dimensional Bayesian models

Projective inference in high-dimensional problems: Prediction and feature selection

Detecting and diagnosing prior and likelihood sensitivity with power-scaling

Contact Info

Product

Resources

About