Heterogeneity estimates in a biased world

Abstract

Meta-analyses typically quantify heterogeneity of results, thus providing information about the consistency of the investigated effect across studies. Numerous heterogeneity estimators have been devised. Past evaluations of their performance typically presumed lack of bias in the set of studies being meta-analysed, which is often unrealistic. The present study used computer simulations to evaluate five heterogeneity estimators under a range of research conditions broadly representative of meta-analyses in psychology, with the aim to assess the impact of biases in sets of primary studies on estimates of both mean effect size and heterogeneity in meta-analyses of continuous outcome measures. To this end, six orthogonal design factors were manipulated: Strength of publication bias; 1-tailed vs. 2-tailed publication bias; prevalence of p-hacking; true heterogeneity of the effect studied; true average size of the studied effect; and number of studies per meta-analysis. Our results showed that biases in sets of primary studies caused much greater problems for the estimation of effect size than for the estimation of heterogeneity. For the latter, estimation bias remained small or moderate under most circumstances. Effect size estimations remained virtually unaffected by the choice of heterogeneity estimator. For heterogeneity estimates, however, relevant differences emerged. For unbiased primary studies, the REML estimator and (to a lesser extent) the Paule-Mandel performed well in terms of bias and variance. In biased sets of primary studies however, the Paule-Mandel estimator performed poorly, whereas the DerSimonian-Laird estimator and (to a slightly lesser extent) the REML estimator performed well. The complexity of results notwithstanding, we suggest that the REML estimator remains a good choice for meta-analyses of continuous outcome measures across varied circumstances.

Communicable Diseases Prevalence among Refugees and Asylum Seekers: Systematic Review and Meta-Analysis

“…This could have created minor discrepancy in the results compared to other studies that considered the disease status as active vs. latent, for example [ 116 ]. Consequently, the results of this review might be subject to a bias and inaccuracies in the pooled prevalence [ 117 ]. However, considering the aim of this review, this expected bias might have little effect on looking at the overall estimates of the diseases in refugee populations at the countries of asylum, although estimating the status of each disease precisely would provide more confidence and rigor to the review.…”

Timing matters: A meta-analysis on the dynamic effect of stress on salivary immunoglobulin

“…Thereby we reduced the risk of false discoveries. Following the recommendations (Hönekopp and Linden, 2022; Langan et al, 2017), Restricted Maximum Likelihood was used as the estimation method, with the exception of model comparisons, which were based on Maximum Likelihood estimation as the compared models included different predictors (Long and Ryoo, 2010).…”

Whoever has will be given more? Exploring the Impact of Non-Linearity on Effect Heterogeneity in Psychological Research.

“…It was applied using an updated version of the MetaPipeX package (quote myself), which uLlizes the metafor package (Viechtbauer,201X) for random effect meta-analyses of mulLple site-level staLsLcs: group means, group and pooled standard deviaLons and unstandardized and standardized mean differences. The model is fit using REML (Hönekopp & Linden, 2022) and the standardized mean differences are calculated as Hedges g (Borenstein et al, 2021). Please refer to the MetaPipeX Update documentaLon for a more detailed descripLon of the staLsLcal models for each parameter (h]ps://github.com/JensFuenderich/MetaPipeX_Update/blob/main/MetaPipeXUpdate_0.0.…”