Capturing the underlying distribution in meta‐analysis: Credibility and tolerance intervals

Abstract

Tolerance intervals provide a bracket intended to contain a percentage (e.g., 80%) of a population distribution given sample estimates of the mean and variance. In random‐effects meta‐analysis, tolerance intervals should contain researcher‐specified proportions of underlying population effect sizes. Using Monte Carlo simulation, we investigated the coverage for five relevant tolerance interval estimators: the Schmidt‐Hunter credibility intervals, a prediction interval, two content tolerance intervals adapted to meta‐analysis, and a bootstrap tolerance interval. None of the intervals contained the desired percentage of coverage at the nominal rates in all conditions. However, the prediction worked well unless the number of primary studies was small (<30), and one of the content tolerance intervals approached nominal levels with small numbers (<20) of primary studies. The bootstrap tolerance interval achieved near nominal coverage if there were sufficient numbers of primary studies (30+) and large enough sample sizes (N ≅ 70) in the included primary studies, although it slightly exceeded nominal coverage with large numbers of large‐sample primary studies. Next, we showed the results of applying the intervals to real data using a set of previously published analyses and provided suggestions for practice. Tolerance intervals incorporate error of estimation into the construction of proper brackets for fractions of population true effects. In many contexts, such intervals approach the desired nominal levels of coverage.

Accuracy and precision of fixed and random effects in meta‐analyses of randomized control trials for continuous outcomes

“…For all univariate meta‐analyses, standard errors and 95% confidence intervals were adjusted following Knapp and Hartung 44 for better control of type I error rates in small samples. Prediction intervals ( PI ) were calculated as where is the standard error of the estimated pooled effect , is the estimated between‐sample variance, is the 97.5 percentile of the t ‐distribution, and gives the number of samples 45 …”

“…where SE b Δ is the standard error of the estimated pooled effect b Δ, b τ 2 is the estimated between-sample variance, t 1,kÀ2 ð Þ is the 97.5 percentile of the t-distribution, and k gives the number of samples. 45…”

Moving from opposition to taking ownership of open science to make discoveries that matter

“…Specific ESM studies (Prem et al, 2018) or meta-analytic evidence (Crawford et al, 2010;Alarcon, 2011) may provide average effect sizes of the bivariate links between workload and exhaustion. However with few exceptions (McCormick et al, 2018), the meta-analyses on this topic speak to the between-person level of analysis, most studies refer to proxies of our focal measures, and random effects estimates typically suggest that a wide range of effect sizes is plausible (Brannick et al, 2021). Hence, based on the literature, a calculation of the standard meta-analytic effect size is imprecise at best.…”