Estimating publication bias in meta-analyses of peer-reviewed studies: A meta-meta-analysis across disciplines and journal tiers

Mathur, Maya B.; VanderWeele, Tyler J.

doi:10.31219/osf.io/p3xyd

Cited by 17 publications

(27 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…If so, publication bias might have been relatively mild in meta-analyses because, in principle, highquality meta-analyses include all studies published in any journal and at any time, so even if elite journals and nascent fields induce severe publication bias by excluding nearly all nonaffirmative results, it is possible that these nonaffirmative results are still eventually published, perhaps in a lower-tier journals, and hence are still included in the meta-analysis. However, additional analyses did not support these hypotheses (Mathur & VanderWeele, 2020a). Instead, preliminary evidence suggested that the key alleviator of publication bias in the meta-analyses may have been their inclusion of "non-headline" results; that is, results that are reported in published papers but that are de-emphasized (e.g., those reported only in secondary or supplemental analyses) and those that meta-analysts obtain through manual calculation or by contacting authors (Mathur & VanderWeele, 2020a).…”

Section: Empirical Benchmarks For Interpreting S(t Q)mentioning

confidence: 96%

“…However, additional analyses did not support these hypotheses (Mathur & VanderWeele, 2020a). Instead, preliminary evidence suggested that the key alleviator of publication bias in the meta-analyses may have been their inclusion of "non-headline" results; that is, results that are reported in published papers but that are de-emphasized (e.g., those reported only in secondary or supplemental analyses) and those that meta-analysts obtain through manual calculation or by contacting authors (Mathur & VanderWeele, 2020a).…”

Section: Empirical Benchmarks For Interpreting S(t Q)mentioning

confidence: 96%

“…To help empirically ground such assessments, we conducted a preregistered meta-meta-analysis to estimate the actual value of η in an objectively chosen sample of meta-analyses across several scientific disciplines. Detailed methods and results are provided in Mathur & VanderWeele (2020a).…”

Section: Empirical Benchmarks For Interpreting S(t Q)mentioning

confidence: 99%

“…Thus, except for Metalab, estimates of publication bias were fairly close to the null and with confidence intervals all bounded below η = 3. We conducted a number of sensitivity analyses to assess the impacts of possible violations of modeling assumptions, all of which yielded similar results (Mathur & VanderWeele, 2020a).…”

Section: Empirical Benchmarks For Interpreting S(t Q)mentioning

confidence: 99%

See 3 more Smart Citations

Sensitivity analysis for publication bias in meta-analyses

Mathur¹,

VanderWeele²

2019

Preprint

Self Cite

105

View full text Add to dashboard Cite

We propose sensitivity analyses for publication bias in meta-analyses. We consider a publication process such that "statistically significant" results are more likely to be published than negative or "nonsignificant" results by an unknown ratio, eta. Our proposed methods also accommodate some plausible forms of selection based on a study's standard error. Using inverse-probability weighting and robust estimation that accommodates non-normal population effects, small meta-analyses, and clustering, we develop sensitivity analyses that enable statements such as: "For publication bias to shift the observed point estimate to the null, 'significant' results would need to be at least 30-fold more likely to be published than negative or 'nonsignificant' results." Comparable statements can be made regarding shifting to a chosen non-null value or shifting the confidence interval. To aid interpretation, we describe empirical benchmarks for plausible values of eta across disciplines. We show that a worst-case meta-analytic point estimate for maximal publication bias under the selection model can be obtained simply by conducting a standard meta-analysis of only the negative and "nonsignificant" studies; this method sometimes indicates that no amount of such publication bias could "explain away" the results. We illustrate the proposed methods using real-life meta-analyses and provide an R package, PublicationBias.

show abstract

Section: Empirical Benchmarks For Interpreting S(t Q)mentioning

confidence: 96%

Section: Empirical Benchmarks For Interpreting S(t Q)mentioning

confidence: 96%

Section: Empirical Benchmarks For Interpreting S(t Q)mentioning

confidence: 99%

Section: Empirical Benchmarks For Interpreting S(t Q)mentioning

confidence: 99%

See 2 more Smart Citations

Sensitivity analysis for publication bias in meta-analyses

Mathur¹,

VanderWeele²

2019

Preprint

Self Cite

105

View full text Add to dashboard Cite

show abstract

“…For more general set of recommendations on how to conduct a rigorous meta-analysis, see a comprehensive guidance paper by Pigott & Polanin (2019). 9 When (1) conducting a prospective or mini meta-analysis and synthesizing (2) a set of studies conducted as registered reports or (3) "non-headline" effects, i.e., effects independent from the focal effect of the paper(Mathur & VanderWeele, 2019b).10 In some analytical contexts, the goal may be just to summarize the results of a finite set of studies without attempting to draw generalizable inferences about unobserved studies (using fixed-effects meta-analytic models).…”

mentioning

confidence: 99%

Neglect of publication bias compromises meta-analyses of educational research

Ropovik¹,

Adamkovič²,

Greger³

2019

Preprint

View full text Add to dashboard Cite

Because negative findings have less chance of getting published, available studies tend to be a biased sample. This leads to a higher prevalence of false positives and the inflation of effect sizes to an unknown degree. To see how meta-analyses in education account for publication bias, we surveyed all meta-analyses published in the last three years in the Review of Educational Research and Educational Research Review. The results show that meta-analyses usually neglect publication bias correction. In the minority of studies adjusting for bias, the outdated trim and fill method was used invariably, and none of the meta-analyses based their conclusions on corrected estimates, rendering the adjustment inconsequential. It is argued that state-of-the-art adjustment (e.g., selection models) should be carried out by default, yet one needs to take into account the uncertainty inherent in any meta-analytic inference under bias. We conclude by providing practical recommendations on dealing with publication bias.

show abstract

Using selection models to assess sensitivity to publication bias: A tutorial and call for more routine use

Maier

VanderWeele

Mathur

2022

Campbell Systematic Reviews

View full text Add to dashboard Cite

In meta-analyses, it is critical to assess the extent to which publication bias might have compromised the results. Classical methods based on the funnel plot, including Egger's test and Trim-and-Fill, have become the de facto default methods to do so, with a large majority of recent meta-analyses in top medical journals (85%) assessing for publication bias exclusively using these methods. However, these classical funnel plot methods have important limitations when used as the sole means of assessing publication bias: they essentially assume that the publication process favors large point estimates for small studies and does not affect the largest studies, and they can perform poorly when effects are heterogeneous. In light of these limitations, we recommend that meta-analyses routinely apply other publication bias methods in addition to or instead of classical funnel plot methods. To this end, we describe how to use and interpret selection models. These methods make the often more realistic assumption that publication bias favors "statistically significant" results and that also directly accommodate effect heterogeneity.Selection models are well-established in the statistics literature and are supported by user-friendly software, yet remain rarely reported in many disciplines. We use previously published meta-analyses to demonstrate that selection models can yield insights that extend beyond those provided by funnel plot methods, suggesting the importance of establishing more comprehensive reporting practices for publication bias assessment.

show abstract

Estimating publication bias in meta-analyses of peer-reviewed studies: A meta-meta-analysis across disciplines and journal tiers

Cited by 17 publications

References 14 publications

Sensitivity analysis for publication bias in meta-analyses

Sensitivity analysis for publication bias in meta-analyses

Neglect of publication bias compromises meta-analyses of educational research

Using selection models to assess sensitivity to publication bias: A tutorial and call for more routine use

Contact Info

Product

Resources

About