Selecting relevant moderators with Bayesian regularized meta‐regression

Abstract

When meta-analyzing heterogeneous bodies of literature, meta-regression can be used to account for potentially relevant between-studies differences. A key challenge is that the number of candidate moderators is often high relative to the number of studies. This introduces risks of overfitting, spurious results, and model non-convergence. To overcome these challenges, we introduce Bayesian Regularized Meta-Analysis (BRMA), which selects relevant moderators from a larger set of candidates by shrinking small regression coefficients towards zero with regularizing (LASSO or horseshoe) priors. This method is suitable when there are many potential moderators, but it is not known beforehand which of them are relevant. A simulation study compared BRMA against state-of-the-art random effects meta-regression using restricted maximum likelihood (RMA). Results indicated that BRMA outperformed RMA on three metrics: BRMA had superior predictive performance, which means that the results generalized better; BRMA was better at rejecting irrelevant moderators, and worse at detecting true effects of relevant moderators, while the overall proportion of Type I and Type II errors was equivalent to RMA. BRMA regression coefficients were slightly biased towards zero (by design), but its residual heterogeneity estimates were less biased than those of RMA. BRMA performed well with as few as 20 studies, suggesting its suitability as a small sample solution. We present free open source software implementations in the R-package pema (for penalized meta-analysis) and in the stand-alone statistical program JASP. An applied example demonstrates the use of the R-package.

Quality Assessment in Meta-Analysis (QuAMA)

“…Besides the level of analysis, meta-analysts may also consider testing for non-linear moderator effects or interaction effects between several quality indicators (e.g., Knop et al, 2023). If many quality indicators are used as separate variables to represent PSQ, then metaanalysts may consider meta-regression modeling approaches that could help them select relevant moderators, non-linear terms, and interactions, such as regularized meta-regression or metaregression based on random forests (Van Lissa, 2020;Van Lissa et al, 2023). Finally, we note that the recent extensions of meta-analytic models to location-scale models or mixed-effects meta-regression models with subgroup-specific (residual) heterogeneities allow meta-analysts to explore direct moderator effects on the heterogeneity estimates (Rubio-Aparicio et al, 2020;…”

Aggregating evidence from conceptual replication studies using the product Bayes factor

“…Finally, when there are many potential variables that cause systematic differences and it is not known beforehand which are relevant, exploratory techniques like random forest meta-analysis and penalized meta-regression can be used to identify relevant moderators (Van Lissa, Van Erp, & Clapper, 2023). However, accounting for moderators requires a relatively high number of observations per moderator, which may not be available.…”