Quantifying uncertainty in the meta-analytic lower bound estimate.

Abstract

In meta-analyses, it is customary to compute a confidence interval for the overall mean effect (ρ̄ or δ̄), but not for the underlying standard deviation (τ) or the lower bound of the credibility value (90%CV), even though the latter entities are often as important to the interpretation as is the overall mean. We introduce 2 methods of computing confidence intervals for the lower bound (Lawless and bootstrap). We compare both methods using 3 lower bound estimators (Schmidt-Hunter, Schmidt-Hunter with k correction, and Morris/Hedges, labeled HOVr/HOVd) in 2 Monte Carlo studies (1 for correlations and 1 for standardized mean differences) and illustrate their application to published meta-analyses. For correlations, we found that HOVr bootstrap confidence intervals yielded coverage greater than 90% across a wide variety of conditions provided that there were at least 25 studies. For the standardized mean difference, all 3 methods produced acceptable results using the bootstrap for the lower bound confidence interval provided that there were at least 20 studies with an average sample size of at least 50. When the number of studies was small (k = 8 or 10), coverage was less than 90% and confidence intervals were very wide. Even with larger numbers of studies, if there was indirect range restriction coupled with a small underlying between-studies variance, the between-studies variance was poorly estimated and coverage of the lower bound suffered. We provide software to allow meta-analysts to compute bootstrap confidence intervals for the estimators described in the article.

Capturing the underlying distribution in meta‐analysis: Credibility and tolerance intervals

“…Because of the additional layer of disattenuation estimation error, we expect that the results in the current paper represent the ceiling for coverage. Brannick, Potter, Teng 67 were interested in the coverage of the lower bound of the credibility interval provided by different estimators. They found similar patterns of results for data with and without reliability artifacts and adjustments, suggesting that we might expect the same pattern of results for our investigation.…”

The Importance of Type I Error Rates When Studying Bias in Monte Carlo Studies in Statistics

“…Equation 5 Values of equation 6closer to one signal that (squared) bias is dominating differences between estimates and a parameter (sampling error is comparatively modest), and values closer to zero that (squared) bias is playing a modest or negligible role (sampling error is comparatively large). For example, 42.2% of the empirical Type I error rates reported in Harring et al (2012) exceeded the bounds of acceptability these authors employed via the Bradley liberal criterion (Bradley, 1978) .025 .075…”

“…The central feature of bias is that it is non-random. The (ˆi  − θ) generated across i = 1, 2,…, J replications in a A critical feature of the above bias measures in Monte Carlo studies are userspecified (arbitrary) cutoffs to distinguish important (non-ignorable) bias from less important (ignorable) bias: Cutoffs for average bias and absolute bias, equations 1and 2, are unique to individual Monte Carlo studies (e.g., Brannick et al, 2019;Yuan et al, 2015), whereas 5% and 10% are typical cutoffs for relative bias and absolute relative bias, equations (3) and (4), although other values are sometimes used.…”

“…Relatedly, many Monte Carlo studies report confidence intervals about parameters of interest. For example, the coverage rate (CR) for confidence intervals about parameters of interest is frequently used as an indicator of standard error bias (Brannick et al, 2019;Chen & Leroux, 2018;Maas & Hox, 2004;Seco et al, 2013;Vallejo et al, 2015). A CR such as 88% for a confidence interval with a nominal coverage probability of .95 often reflects negatively-biased standard errors, whereas a CR of 98% reflects positively-biased standard errors.…”

“…The same logic can be applied to other versions of bias reported in Monte Carlo studies. For example, McNeish and Harring (2017) cited Bradley (1978) in using cutoffs of 92% and 98% for CRs about parameters in 95% confidence intervals, with CRs outside these values treated as reflecting biased standard errors. Marrying Type I error rates with bias means the bias of standard errors associated with 92% ≤ CR ≤ 98% should be treated as zero, whereas the bias of standard errors associated with CR < 92% or CR > 98% is not zero.…”