Adaptive random neighbourhood informed Markov chain Monte Carlo for high-dimensional Bayesian variable selection

Abstract

We introduce a framework for efficient Markov chain Monte Carlo algorithms targeting discrete-valued high-dimensional distributions, such as posterior distributions in Bayesian variable selection problems. We show that many recently introduced algorithms, such as the locally informed sampler of Zanella (J Am Stat Assoc 115(530):852–865, 2020), the locally informed with thresholded proposal of Zhou et al. (Dimension-free mixing for high-dimensional Bayesian variable selection, 2021) and the adaptively scaled individual adaptation sampler of Griffin et al. (Biometrika 108(1):53–69, 2021), can be viewed as particular cases within the framework. We then describe a novel algorithm, the adaptive random neighbourhood informed sampler, which combines ideas from these existing approaches. We show using several examples of both real and simulated data-sets that a computationally efficient point-wise implementation (PARNI) provides more reliable inferences on a range of variable selection problems, particularly in the very large p setting.

Adaptive Computational Methods for Bayesian Variable Selection

“…Some recently developed algorithms for high-dimensional variable selection use an independent approximation to the posterior distribution. In MCMC methods, the ASI algorithm , and its development to ARNI and PARNI (Liang et al, 2022) use an independent approximation. Similarly, Ray and Szabó (2022) study variational inference for Bayesian variable selection where the variational distribution is independent across the variables.…”

“…, γ (N ) from the posterior distribution p(γ|Data). Unlike the first approach, this leads to unbiased estimates of posterior quantities but it can be computationally expensive to generate a representative sample (Zanella and Roberts, 2019;Zhou et al, 2022;Liang et al, 2022).…”

“…This approach has good theoretical properties (Castillo et al, 2015) and predictive performance (Porwal and Raftery, 2022). Recent work has provided efficient algorithms for use in high-dimensional problems involving tens of thousands of regressors, including Markov chain Monte Carlo (MCMC) methods such as Importance Tempering (Zanella and Roberts, 2019), ASI , LIT (Zhou et al, 2022) and PARNI (Liang et al, 2022), or stochastic search methods such as SVEN (Li et al, 2023). We will follow the spike-and-slab approach using MCMC but the methods could be extended to approximations of the posterior from a spike-and-slab prior found using different computational methods such as stochastic search.…”

Searching in ultra high dimensional sparse model spaces: New performance tests and boosting Gibbs sampling algorithms

“…ASI is able to adapt to the importance of each candidate covariate and propose multiple swaps using MCMC methods with the construction of adaptive random neighbourhood informed proposals. In a more recent paper, Liang et al (2022) thoroughly analysed the ASI algorithm, showing that ASI is a random neighbourhood sampler with a second stage that is a random walk proposal. Building on this, Liang et al (2022) introduced a new algorithm that uses an informed withinneighbourhood proposal in the second stage.…”

Adaptive MCMC for Bayesian Variable Selection in Generalised Linear Models and Survival Models

“…The tuning parameter ζ ∈ (ϵ, 1 − ϵ) denotes the non-informative jumping probability. Two different methods for adapting ζ are provided in [32]. The function g is a balancing function that satisfies g(x) = xg(1/x).…”