Model selection in Medical Research: A simulation study comparing Bayesian Model Averaging and Stepwise Regression

Abstract

BackgroundAutomatic variable selection methods are usually discouraged in medical research although we believe they might be valuable for studies where subject matter knowledge is limited. Bayesian model averaging may be useful for model selection but only limited attempts to compare it to stepwise regression have been published. We therefore performed a simulation study to compare stepwise regression with Bayesian model averaging.MethodsWe simulated data corresponding to five different data generating processes and thirty different values of the effect size (the parameter estimate divided by its standard error). Each data generating process contained twenty explanatory variables in total and had between zero and two true predictors. Three data generating processes were built of uncorrelated predictor variables while two had a mixture of correlated and uncorrelated variables. We fitted linear regression models to the simulated data. We used Bayesian model averaging and stepwise regression respectively as model selection procedures and compared the estimated selection probabilities.ResultsThe estimated probability of not selecting a redundant variable was between 0.99 and 1 for Bayesian model averaging while approximately 0.95 for stepwise regression when the redundant variable was not correlated with a true predictor. These probabilities did not depend on the effect size of the true predictor. In the case of correlation between a redundant variable and a true predictor, the probability of not selecting a redundant variable was 0.95 to 1 for Bayesian model averaging while for stepwise regression it was between 0.7 and 0.9, depending on the effect size of the true predictor. The probability of selecting a true predictor increased as the effect size of the true predictor increased and leveled out at between 0.9 and 1 for stepwise regression, while it leveled out at 1 for Bayesian model averaging.ConclusionsOur simulation study showed that under the given conditions, Bayesian model averaging had a higher probability of not selecting a redundant variable than stepwise regression and had a similar probability of selecting a true predictor. Medical researchers building regression models with limited subject matter knowledge could thus benefit from using Bayesian model averaging.

Education Attainment is Associated With Patient-reported Outcomes: Findings From the Swedish Hip Arthroplasty Register

“…Bayesian model averaging identified the significant predictors of each outcome parameter [12]. Bayesian model averaging does not base inference on a single model, but on a combination of models; thus it avoids model selection bias and has good specificity without compromising sensitivity [9]. Decisions regarding the importance of a predictor in the model are based on the estimated posterior probability.…”

“…The index reflected a weighted population-based representation of the five dimensions (mobility, self-care, usual activities, pain/discomfort, and anxiety/depression), and the EQ VAS was an overview of the patient's impression of their HRQoL on that day. Significant variables (minimum of 0.50 posterior probability thresholds) were included in the final linear regression analyses to determine the effect of each preoperative patient factor on the outcome parameter in question [9,15].…”

Contributions of Caucasian-associated bone mass loci to the variation in bone mineral density in Vietnamese population

“…We also observed a significant association between rs7543680 305 (ZBTB40 gene) and BMD, and to our knowledge, this is the second find- to yield misleading and/or false positive results [40]. In this study, we 335 used the Bayesian model average approach which has been shown to 336 have better performance than the traditional stepwise method [20,21] 337 in terms of identification of relevant variables in the regression model.…”

“…In this study, we used the Bayesian model average (BMA) ap-174 proach [19] to identify genetic variants that were associated with BMD. 175 This approach has been shown to have superior performance compared 176 to "traditional" approaches such as stepwise regression [20,21]. If M = 177 (M 1 , M 2 , M 3 , …, M k ) denotes the set of all possible models considered, the idea is to find the "optimal" models in that space.…”

Prediction of Influential Operational Compost Parameters for Monitoring Composting Process

“…The criteria and stepwise estimation were used to add and remove fields. The p-value used was 0.05 (Genell et al, 2010).…”