Reproducibility of individual effect sizes in meta-analyses in psychology

Maassen, Esther; Assen, Marcel A.L.M. van; Nuijten, Michèle B.; Olsson-Collentine, Anton; Wicherts, Jelte M.

doi:10.1371/journal.pone.0233107

Cited by 58 publications

(61 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The scientific methods for meta-analysis are well-established (Borenstein et al, 2009;Cooper et al, 2019;Glass, 2015;. Other recommended practices go further, also recommending sharing data and code for quality control, reproducibility of results, and updating earlier meta-analyses (Button et al, 2013;Gurevitch et al, 2018;Lakens et al, 2016;Maassen et al, 2020;Polanin et al, 2020). However, even the well-established methods for research synthesis are not universally followed.…”

Section: Resultsmentioning

confidence: 99%

Statistical Significance Filtering Overestimates Effects and Impedes Falsification: A Critique of Endsley (2019)

Bakdash¹,

Marusich²,

Kenworthy³

et al. 2020

Preprint

View full text Add to dashboard Cite

Whether in meta-analysis or single experiments, selecting results based on statistical significance leads to overestimated effect sizes, impeding falsification. We critique a quantitative synthesis that used significance to score and select “relevant” previously published effects for situation awareness-performance associations (Endsley, 2019). How much does selection using statistical significance quantitatively impact results in a meta-analytic context? We evaluate and compare results using significance-filtered effects versus analyses with all effects as-reported. Endsley reported high predictiveness scores and large positive mean correlations, but used atypical methods: the hypothesis was used to select papers and effects. Papers were assigned the maximum predictiveness scores if they contained at-least-one significant effect, yet most papers reported multiple effects, and the number of non-significant effects did not impact the score. Thus, the predictiveness score was rarely less than the maximum. In addition, only significant effects were included in Endsley’s quantitative synthesis. Filtering excluded half of all reported effects, with guaranteed minimum effect sizes based on sample size. Results for filtered compared to as-reported effects clearly diverged. Compared to the mean of as-reported effects, the filtered mean was overestimated by 56%. Furthermore, 92% (or 222 out of 241) of the as-reported effects were below the mean of filtered effects. We conclude outcome-dependent selection of effects is circular, predetermining results, impeding falsification, and running contrary to the purpose of meta-analysis. Instead of using significance to score and filter effects, meta-analyses should follow established research practices.

show abstract

Section: Resultsmentioning

confidence: 99%

Statistical Significance Filtering Overestimates Effects and Impedes Falsification: A Critique of Endsley (2019)

Bakdash¹,

Marusich²,

Kenworthy³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…We chose to check RECALL performance first, given that RECALL as outcome is rather clear (i.e., what constitutes RECALL opposed to what constitutes TRANSFER). For effect size recalculation, we followed the workflow of Maassen et al ( 2020 ; their Figure 1, page 5): We identified appropriate (composite) effects in primary studies based on available information in the meta-analysis; when more than one (composite) effect calculation was possible, we calculated multiple logically possible effects and chose the effect size closest to the primary effect reported in the meta-analysis.…”

Section: Guidelines and Inconsistencies In Xie’s Et Al’s ( mentioning

confidence: 99%

“…Such problems are not confined to Xie’s and colleagues’ meta-analysis (cf. Maassen et al 2020 ); nevertheless, they undermine reproducibility, given that it was often unclear to us exactly which effect sizes were selected, whether and how they were combined, and exactly which calculations steps and formula were applied.…”

Section: Guidelines and Inconsistencies In Xie’s Et Al’s ( mentioning

confidence: 99%

“…In this respect, we noticed several shortcomings in Xie’s and colleagues’ meta-analysis. We acknowledge that transparency and reproducibility issues in general also plague other meta-analyses and have proven to be more (e.g., Gøtzsche et al 2007 ) or less consequential (e.g., Jones et al 2005 ; Maassen et al 2020 ). We further acknowledge that following meta-analytical guidelines closely improves meta-analytical quality (e.g., Schalken and Rietbergen 2017 ).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Null and Void? Errors in Meta-analysis on Perceptual Disfluency and Recommendations to Improve Meta-analytical Reproducibility

2021

View full text Add to dashboard Cite

In the 2018 meta-analysis of Educational Psychology Review entitled “Null effects of perceptual disfluency on learning outcomes in a text-based educational context” by Xie, Zhou, and Liu, we identify some errors and inconsistencies in both the methodological approach and the reported results regarding coding and effect sizes. While from a technical point of view the meta-analysis aligns with current meta-analytical guidelines (e.g., PRISMA) and conforms to general meta-analytical requirements (e.g., considering publication bias), it exemplifies certain insufficient practices in the creation and review of meta-analysis. We criticize the lack of transparency and negligence of open-science practices in the generation and reporting of results, which complicate evaluation of the meta-analytical reproducibility, especially given the flexibility in subjective choices regarding the analytical approach and the flexibility in creating the database. Here we present a framework applicable to pre- and post-publication review on improving the Methods Reproducibility of meta-analysis. Based on considerations of the transparency and openness (TOP)-guidlines (Nosek et al. Science 348: 1422–1425, 2015), the Reproducibility Enhancement Principles (REP; Stodden et al. Science 354:1240–1241, 2016), and recommendations by Lakens et al. (BMC Psychology 4: Article 24, 2016), we outline Computational Reproducibility (Level 1), Computational Verification (Level 2), Analysis Reproducibility (Level 3), and Outcome Reproducibility (Level 4). Applying reproducibility checks to TRANSFER performance as the chosen outcome variable, we found Xie’s and colleagues’ results to be (rather) robust. Yet, regarding RECALL performance and the moderator analysis, the identified problems raise doubts about the credibility of the reported results.

show abstract

“…The scientific methods for meta-analysis are well-established ( Borenstein et al, 2009 ; Koricheva et al, 2013 ; Glass, 2015 ; Cooper et al, 2019 ). Other recommended practices go further, also recommending sharing data and code for quality control, reproducibility of results, and updating earlier meta-analyses ( Button et al, 2013 ; Lakens et al, 2016 ; Gurevitch et al, 2018 ; Maassen et al, 2020 ; Polanin et al, 2020 ). However, even the well-established methods for research synthesis are not universally followed.…”

Section: Discussionmentioning

confidence: 99%

Statistical Significance Filtering Overestimates Effects and Impedes Falsification: A Critique of Endsley (2019)

Bakdash¹,

Marusich²,

Kenworthy³

et al. 2020

Front. Psychol.

View full text Add to dashboard Cite

Whether in meta-analysis or single experiments, selecting results based on statistical significance leads to overestimated effect sizes, impeding falsification. We critique a quantitative synthesis that used significance to score and select previously published effects for situation awareness-performance associations (Endsley, 2019). How much does selection using statistical significance quantitatively impact results in a meta-analytic context? We evaluate and compare results using significance-filtered effects versus analyses with all effects as-reported. Endsley reported high predictiveness scores and large positive mean correlations but used atypical methods: the hypothesis was used to select papers and effects. Papers were assigned the maximum predictiveness scores if they contained at-least-one significant effect, yet most papers reported multiple effects, and the number of non-significant effects did not impact the score. Thus, the predictiveness score was rarely less than the maximum. In addition, only significant effects were included in Endsley’s quantitative synthesis. Filtering excluded half of all reported effects, with guaranteed minimum effect sizes based on sample size. Results for filtered compared to as-reported effects clearly diverged. Compared to the mean of as-reported effects, the filtered mean was overestimated by 56%. Furthermore, 92% (or 222 out of 241) of the as-reported effects were below the mean of filtered effects. We conclude that outcome-dependent selection of effects is circular, predetermining results and running contrary to the purpose of meta-analysis. Instead of using significance to score and filter effects, meta-analyses should follow established research practices.

show abstract

Reproducibility of individual effect sizes in meta-analyses in psychology

Cited by 58 publications

References 30 publications

Statistical Significance Filtering Overestimates Effects and Impedes Falsification: A Critique of Endsley (2019)

Statistical Significance Filtering Overestimates Effects and Impedes Falsification: A Critique of Endsley (2019)

Null and Void? Errors in Meta-analysis on Perceptual Disfluency and Recommendations to Improve Meta-analytical Reproducibility

Statistical Significance Filtering Overestimates Effects and Impedes Falsification: A Critique of Endsley (2019)

Contact Info

Product

Resources

About