A novel Bayesian approach for variable selection in linear regression models

Abstract

We propose a novel Bayesian approach to the problem of variable selection in multiple linear regression models. In particular, we present a hierarchical setting which allows for direct specification of a-priori beliefs about the number of nonzero regression coefficients as well as a specification of beliefs that given coefficients are nonzero. To guarantee numerical stability, we adopt a g-prior with an additional ridge parameter for the unknown regression coefficients. In order to simulate from the joint posterior distribution an intelligent random walk Metropolis-Hastings algorithm which is able to switch between different models is proposed. Testing our algorithm on real and simulated data illustrates that it performs at least on par and often even better than other well-established methods. Finally, we prove that under some nominal assumptions, the presented approach is consistent in terms of model selection. 45However, the g-prior depends on the inverse of the empirical covariance matrix of the selected predictors. This matrix is singular if the number of selected covariates is greater than the number of observations n and, further, may be almost rank deficient given that the predictors are highly correlated. To overcome this problem Wang et al. (2015) replaced the classical inverse with the Moore-Penrose generalized inverse and thus ended up with the so-called gsg-prior (see West (2003)). In contrast to them, we adopt a g-prior 50 with an additional ridge parameter for the unknown regression coefficients to guarantee nonsingularity of the empirical covariance matrix. This modification of the classical g-prior was first proposed by Gupta and Ibrahim (2007) and further investigated by Baragatti and Pommeret (2012).Finally, in Section 2.2 we state that our approach is consistent in terms of model selection according to the consistency definition given by Fernández et al. (2001). The proof of this result is deferred to the 55 appendix. Moreover, in Section 3, we evaluate our approach on the basis of real and simulated data and compare the results with the already described shrinkage methods. We show that our approach performs at least on par and often better than the comparative methods.

A Bayesian approach to understanding the key factors influencing temporal variability in stream water quality – a case study in the Great Barrier Reef catchments

“…The hierarchical modelling framework was implemented in JAGS (Plummer, 2013a), using the package rjags in R (Plummer, 2013b;R Core Team, 2013), which enabled both the estimation of parameter values from prior distributions with MCMC and the generation of model-averaged predictions. The MCMC sampling had three parallel chains with 25 000 iterations for each chain.…”

“…1.00 and 0.34 for pre-event NDVI and event NDVI, respectively, for DON in cluster 2). The BMA can handle the collinearity by shrinking the posterior distribution of inclusion probability of one of the correlated variables towards zero (Nakagawa and Freckleton, 2011;Posch et al, 2020;Walker, 2019). This shrinkage effect leads to a lower posterior probability of a more complex model that includes correlated variables because each extra predictor dilutes the prior density of the existing predictor that it correlates with.…”

Variable Selection Using Nearest Neighbor Gaussian Processes

“…• Bayes linear selection (BLS). Recently, Posch et al (2020) proposed a Bayesian variable selection approach for linear regression models. Since our approach shares some common features (e.g.…”

“…Following Posch et al (2020) to specify the proposal for the random set A at first a Bernoulli distributed random variable c h is introduced…”

“…In case that the dependencies of interest are purely linear simple linear regression models are sufficient for the analysis. There are many well-established approaches to perfrom variable selection in linear regression models (Tibshirani, 1996;Fan and Li, 2001;Zou and Hastie, 2005;Zou, 2006;Park and Casella, 2008;Ročková and George, 2014;Alhamzawi and Taha Mohammad Ali, 2018;Posch et al, 2020). However, in many practical applications linearity does not hold and more sophisticated methods are required.…”