Reproducibility of Individual Effect Sizes in Meta-Analyses in Psychology

Maassen, Esther; Assen, Marcel A.L.M. van; Nuijten, Michèle B.; Olsson-Collentine, Anton; Wicherts, Jelte M.

doi:10.31234/osf.io/g5ryh

Cited by 12 publications

(14 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Often, the meta-analysts calculated the effect size and its SE of a primary study themselves. Analyzing data and reporting results are error prone (see, e.g., Bakker and Wicherts 2011;Gøtzsche et al 2007;Maassen et al 2020;Mathes et al 2017;Nuijten et al 2016;Petrocelli et al 2012). To minimize the risk of copying erroneously calculated or reported effect sizes and SEs, we calculated the effect sizes and SEs ourselves using data reported in the meta-analysis, where possible.…”

Section: Methodsmentioning

confidence: 99%

Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis

et al. 2020

Self Cite

View full text Add to dashboard Cite

In this meta-study, we analyzed 2442 effect sizes from 131 meta-analyses in intelligence research, published from 1984 to 2014, to estimate the average effect size, median power, and evidence for bias. We found that the average effect size in intelligence research was a Pearson’s correlation of 0.26, and the median sample size was 60. Furthermore, across primary studies, we found a median power of 11.9% to detect a small effect, 54.5% to detect a medium effect, and 93.9% to detect a large effect. We documented differences in average effect size and median estimated power between different types of intelligence studies (correlational studies, studies of group differences, experiments, toxicology, and behavior genetics). On average, across all meta-analyses (but not in every meta-analysis), we found evidence for small-study effects, potentially indicating publication bias and overestimated effects. We found no differences in small-study effects between different study types. We also found no convincing evidence for the decline effect, US effect, or citation bias across meta-analyses. We concluded that intelligence research does show signs of low power and publication bias, but that these problems seem less severe than in many other scientific fields.

show abstract

Section: Methodsmentioning

confidence: 99%

Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…Achieving reproducibility is a basic foundation of credibility, and yet many efforts to test reproducibility reveal success rates below 100%. For example, Artner and colleagues (2020) successfully reproduced just 70% of 232 findings, and 18 of those only after deviating from the reported analysis in the paper (see also Hardwicke et al, 2018Maassen et al, 2020;Nuijten et al, 2016). Whereas an outcome reproducibility failure suggests that the original result may be wrong, a process reproducibility failure merely indicates that it cannot be verified.…”

Section: Reproducibilitymentioning

confidence: 99%

Replicability, Robustness, and Reproducibility in Psychological Science

Nosek

Hardwicke

Moshontz

et al. 2022

Annu. Rev. Psychol.

403

244

View full text Add to dashboard Cite

Replication—an important, uncommon, and misunderstood practice—is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress. Expected final online publication date for the Annual Review of Psychology, Volume 73 is January 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

show abstract

“…Moreover, a range of tools can be used in this framework to deal with heterogeneous data in an integrated view, and even check for bias in the field as a whole (e.g., Sterne et al, 2005). (Ioannidis, 2016), and recent investigations into the transparency and reproducibility of meta-analyses revealed considerable issues (Polanin, Hennessy & Tsuji, 2020;Maassen et al, 2020). This makes sense: no tool can force its handler to use it wisely.…”

Section: Qualitative Reviewsmentioning

confidence: 99%

A meta-analytic approach to evaluating the explanatory adequacy of theories

Cristià¹,

Tsuji²,

Bergmann³

2020

Preprint

View full text Add to dashboard Cite

Theories are a key part of the scientific process. How should we evaluate theories against evidence? The two traditional and most widespread approaches use single studies and qualitative reviews to evaluate theories, and more recently large-scale replications are used. We argue here that none of these approaches fits in with cumulative science tenets. We propose instead the use of online and transparent Community-Augmented Meta-Analyses (CAMAs). These are cumulative and open because they are built using all available data, with techniques available to prevent data overfitting and to statistically model methodological "noise". We provide counter-arguments for the most common complaints against using meta-analyses for theory evaluation. Finally, we come to conclude that CAMAs will highlight areas of uncertainty better than the alternatives, and could trigger a much needed shift towards a cumulative mindset both with respect to theory and data, leading us to do experiments and qualitative reviews differently.

show abstract

Reproducibility of Individual Effect Sizes in Meta-Analyses in Psychology

Cited by 12 publications

References 20 publications

Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis

Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis

Replicability, Robustness, and Reproducibility in Psychological Science

A meta-analytic approach to evaluating the explanatory adequacy of theories

Contact Info

Product

Resources

About