Evaluation of heterogeneity and heterogeneity interval estimators in random-effects meta-analysis of the standardized mean difference in education and psychology.

Abstract

Meta-analyses are conducted to synthesize the quantitative results of related studies. The random-effects meta-analysis model is based on the assumption that a distribution of true effects exists in the population. This distribution is often assumed to be normal with a mean and variance. The population variance, also called heterogeneity, can be estimated numerous ways. Research exists comparing subsets of heterogeneity estimators over limited conditions. Additionally, heterogeneity is a parameter estimated with uncertainty. Various methods exist for heterogeneity interval estimation, and similar to heterogeneity estimators, these evaluations are limited. The current simulation study examined the performance of Bayesian (with 5 prior specifications) and non-Bayesian estimators over conditions found after a review of meta-analyses of the standardized mean difference in education and psychology research. Three simulation conditions were varied: (a) number of effect sizes per meta-analysis, (b) true heterogeneity, and (c) sample size per effect size within each meta-analysis. Estimators were evaluated over average bias and means square error. Methods of interval estimation were then evaluated with the estimators found to operate optimally. Interval estimators were evaluated based on coverage probability, interval width, and coverage of the estimated value. Overall, the Paule and Mandel estimator, in conjunction with the Jackson method of interval estimation, is recommended if no knowledge exists with regards to the expected value of heterogeneity when synthesizing the standardized mean difference effect size. If heterogeneity is expected to be small (e.g., < .075), then REML with the profile likelihood interval estimator is recommended. Sensitivity analysis evaluating differences in substantive conclusions with a suite of heterogeneity estimators, such as Paule and Mandel, REML, and Hedges and Olkin, is recommended.

The Fear of COVID-19 Scale: A Reliability Generalization Meta-Analysis

“…As random-effects models were assumed, the reliability coefficients were weighted by the inverse variance method, where the variance is the sum of the within-study and the between-studies variances. Between-study variance, τ 2 , was estimated using the Paule and Mandel estimator (Boedeker & Henson, 2020). The 95% confidence interval around each overall reliability estimate was computed with the improved method proposed by Hartung (1999).…”

Comparison of commonly used methods in random effects meta-analysis: application to preclinical data in drug discovery research

“…Vague priors express our a priori uncertainty on the subject and permit the data to determine the posterior while an informative prior represents high certainty in the range of parameter values and dominates the posterior, requiring more data to shift that credibility. 26 In many meta-analyses, a vague prior distribution is chosen especially for the overall effect such that the inference on the primary interest is derived based on the observed data alone. 27 A wide normal distribution is a particularly convenient and commonly used prior in meta-analysis with most common effect measures such as mean differences, standardised mean differences and (log) odds, risk and HRs.…”

“…Uniform distribution, half normal and half Cauchy are convenient prior distributions for the heterogeneity parameter. 26 Alternatively, inverse gamma prior distribution is recommended for heterogeneity as it is known to result in better convergence and bias. 26 29 However, it should be noted that the performance of estimators may vary depending on the type of effect measure of interest.…”

“… 26 Alternatively, inverse gamma prior distribution is recommended for heterogeneity as it is known to result in better convergence and bias. 26 29 However, it should be noted that the performance of estimators may vary depending on the type of effect measure of interest. Hence, sensitivity analysis is recommended to assess the impact of different prior specifications for τ 2 .…”

Age and Moral Foundations: A Meta-Analytic Approach

“…The effect sizes were weighted by the inverse variance method, where the variance is the sum of the withinstudy and the between-studies variances. Betweenstudy variance, τ 2 , was estimated using the Paule and Mandel estimator (Boedeker and Henson, 2020). The 95% confidence interval around each overall reliability estimate was computed with the improved method proposed by Hartung (1999).…”