How to detect publication bias in psychological research? A comparative evaluation of six statistical methods

Renkewitz, Frank; Keiner, Melanie

doi:10.31234/osf.io/w94ep

Cited by 27 publications

(52 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The contour-enhanced funnel plots (Figure 4) did not show any over-representation of marginally significant or just-significant effects in any case. However, it should be acknowledged that STATISTICAL POWER IN GERONTOLOGY 21 many tests for publication bias, including trim-and-fill (Duvall & Tweedie, 2000), p-curve Simmons, 2014), andp-uniform (van Assen, van Aert, &Wicherts, 2015) are inaccurate when true effect sizes are heterogeneous (Renkewitz & Keiner, 2018; van Aert, Wicherts, & van Assen, 2016), as is almost certainly the case in the current study due to the wide range of meta-analyses included.…”

Section: Discussionmentioning

confidence: 71%

Effect Size Interpretation, Sample Size Calculation, and Statistical Power in Gerontology

Brydges¹

2019

Preprint

View full text Add to dashboard Cite

Background and Objectives: Researchers typically use Cohen’s guidelines of Pearson’s r = .10, .30, and .50, and Cohen’s d = 0.20, 0.50, and 0.80 to interpret observed effect sizes as small, medium, or large, respectively. However, these guidelines were not based on quantitative estimates, and are only recommended if field-specific estimates are unknown. The current study investigated the distribution of effect sizes in both individual differences research and group differences research in gerontology to provide estimates of effect sizes in the field.Research Design and Methods: Effect sizes (Pearson’s r, Cohen’s d, and Hedges’ g) were extracted from meta-analyses published in ten top-ranked gerontology journals. The 25th, 50th, and 75th percentile ranks were calculated for Pearson’s r (individual differences) and Cohen’s d or Hedges’ g (group differences) values as indicators of small, medium, and large effects. A priori power analyses were conducted for sample size calculations given the observed effect size estimates.Results: Effect sizes of Pearson’s r = .12, .20, and .32 for individual differences research and Hedges’ g = 0.16, 0.38, and 0.76 for group differences research were interpreted as small, medium, and large effects in gerontology. Discussion and Implications: Cohen’s guidelines appear to overestimate effect sizes in gerontology. Researchers are encouraged to use Pearson’s r = .10, .20, and .30, and Cohen’s d or Hedges’ g = 0.15, 0.40, and 0.75 to interpret small, medium, and large effects in gerontology, and recruit larger samples.

show abstract

Section: Discussionmentioning

confidence: 71%

Effect Size Interpretation, Sample Size Calculation, and Statistical Power in Gerontology

Brydges¹

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Lastly, there is considerable inherent uncertainty associated with the effect size estimates NEGLECT OF PUBLICATION BIAS IN EDUCATIONAL RESEARCH (Carter et al, 2019, Renkewitz & Keiner, 2018van Assen et al, 2015), also leading to lower (sometimes very low) statistical power for detecting more subtle effect sizes. 7…”

Section: Discussionmentioning

confidence: 99%

“…It is indeed difficult to counter that grim feeling. If there is bias in the literature, it has been well demonstrated that unadjusted meta-analysis tends to detect an effect even if there is none, and the current adjustments are imperfect (Carter et al, 2019;Renkewitz & Keiner, 2018).…”

Section: Discussionmentioning

confidence: 99%

Neglect of publication bias compromises meta-analyses of educational research

Ropovik¹,

Adamkovič²,

Greger³

2019

Preprint

View full text Add to dashboard Cite

Because negative findings have less chance of getting published, available studies tend to be a biased sample. This leads to a higher prevalence of false positives and the inflation of effect sizes to an unknown degree. To see how meta-analyses in education account for publication bias, we surveyed all meta-analyses published in the last three years in the Review of Educational Research and Educational Research Review. The results show that meta-analyses usually neglect publication bias correction. In the minority of studies adjusting for bias, the outdated trim and fill method was used invariably, and none of the meta-analyses based their conclusions on corrected estimates, rendering the adjustment inconsequential. It is argued that state-of-the-art adjustment (e.g., selection models) should be carried out by default, yet one needs to take into account the uncertainty inherent in any meta-analytic inference under bias. We conclude by providing practical recommendations on dealing with publication bias.

show abstract

“…We will use the weightr package (Coburn & Vevea, 2019) in the R platform. Selection model approaches outperform other traditional methods (e.g., trim-and-fill method; Carter et al, 2019) Notably, a recent simulation bias detection study found that no method of publication bias outperformed the others, and therefore, it is best to choose a method-based upon the data (Renkewitz & Keiner, 2018). Thus, we have outlined many methods to account for that possibility.…”

Section: Publication Biasmentioning

confidence: 99%

“…Thus, we have outlined many methods to account for that possibility. However, additional methods may be used based on heterogeneity or homogeneity of effect sizes once analyses begin (Renkewitz & Keiner, 2018).…”

Section: Publication Biasmentioning

confidence: 99%

Discrimination, Life Stress, and Mental Health among Muslims: A Preregistered Systematic Review and Meta-Analysis

Kathawalla¹,

Syed²

2019

Preprint

View full text Add to dashboard Cite

Stage 1 Registered Report: How is the current context of Islamophobia, anti-Muslim prejudice, and discrimination, associated with Muslim mental health in Western countries? A body of evidence suggests that experiences of discrimination and racism are associated with negative mental health outcomes for marginalized, minority populations (e.g., Paradies et al., 2015). Studies specifically with Muslim populations in Western countries have found mostly similar findings of an association between discrimination and negative mental health outcomes (Samari, Alcalá, & Sharif, 2018). The varied results in the literature calls for using statistical rigor and synthesis to provide clarity of the etiology of psychopathology in the Muslim community. In this study, we plan to conduct meta-analyses of the associations of both perceived discrimination and mental health and life stressors and mental health for Muslims living in Western countries. We aim to contribute to the literature on the experiences of Muslims and to the debate on the contribution of discrimination vs. life stressors on the development of mental health using meta-analytic methods. For added specificity, we examine both perceived individual discrimination and group discrimination and positive and negative mental health outcomes. We also plan to examine various study-related (e.g., publication bias), demographic, and identity-related moderators in these associations. In the current sociopolitical climate, this study is an important step to better serve the needs of the growing global Muslim community.

show abstract

How to detect publication bias in psychological research? A comparative evaluation of six statistical methods

Cited by 27 publications

References 39 publications

Effect Size Interpretation, Sample Size Calculation, and Statistical Power in Gerontology

Effect Size Interpretation, Sample Size Calculation, and Statistical Power in Gerontology

Neglect of publication bias compromises meta-analyses of educational research

Discrimination, Life Stress, and Mental Health among Muslims: A Preregistered Systematic Review and Meta-Analysis

Contact Info

Product

Resources

About