The prevalence of statistical reporting errors in psychology (1985–2013)

Nuijten, Michèle B.; Hartgerink, Chris; Assen, Marcel A.L.M. van; Epskamp, Sacha; Wicherts, Jelte M.

doi:10.3758/s13428-015-0664-2

Cited by 427 publications

(619 citation statements)

References 33 publications

Supporting

Mentioning

581

Contrasting

Unclassified

Order By: Relevance

“…This can be an issue if multiple settings can be justified and/or if some settings leading to significant outcomes are actually less justified than alternative settings. Publishing scripts will also provide more information on potential statistical errors (Bakker and Wicherts, 2001;Nuijten et al, 2016).…”

Section: Publish All Analysis Scripts With Analysis Settingsmentioning

confidence: 99%

When null hypothesis significance testing is unsuitable for research: a reassessment

Szűcs

Ioannidis

2016

Preprint

107

View full text Add to dashboard Cite

Null hypothesis significance testing (NHST) has several shortcomings that are likely contributing factors behind the widely debated replication crisis of (cognitive) neuroscience, psychology, and biomedical science in general. We review these shortcomings and suggest that, after sustained negative experience, NHST should no longer be the default, dominant statistical practice of all biomedical and psychological research. If theoretical predictions are weak we should not rely on all or nothing hypothesis tests. Different inferential methods may be most suitable for different types of research questions. Whenever researchers use NHST they should justify its use, and publish pre-study power calculations and effect sizes, including negative findings. Hypothesis-testing studies should be pre-registered and optimally raw data published. The current statistics lite educational approach for students that has sustained the widespread, spurious use of NHST should be phased out.

show abstract

Section: Publish All Analysis Scripts With Analysis Settingsmentioning

confidence: 99%

When null hypothesis significance testing is unsuitable for research: a reassessment

Szűcs

Ioannidis

2016

Preprint

107

View full text Add to dashboard Cite

show abstract

“…A manual validity check for statcheck proved that the software is valid for extracting APA style reported test results [1]. However, it does not extract results that are not in line with what the APA prescribes.…”

Section: Usage Notesmentioning

confidence: 99%

688,112 Statistical Results: Content Mining Psychology Articles for Statistical Test Results

Hartgerink

2016

Data

View full text Add to dashboard Cite

Abstract:In this data deposit, I describe a dataset that is the result of content mining 167,318 published articles for statistical test results reported according to the standards prescribed by the American Psychological Association (APA). Articles published by the APA, Springer, Sage, and Taylor & Francis were included (mining from Wiley and Elsevier was actively blocked). As a result of this content mining, 688,112 results from 50,845 articles were extracted. In order to provide a comprehensive set of data, the statistical results are supplemented with metadata from the article they originate from. The dataset is provided in a comma separated file (CSV) in long-format. For each of the 688,112 results, 20 variables are included, of which seven are article metadata and 13 pertain to the individual statistical results (e.g., reported and recalculated p-value). A five-pronged approach was taken to generate the dataset: (i) collect journal lists; (ii) spider journal pages for articles; (iii) download articles; (iv) add article metadata; and (v) mine articles for statistical results. All materials, scripts, etc. are available at https://github.com/chartgerink/2016statcheck_data and preserved at http://dx.doi.org/10.5281/zenodo.59818. SummaryIn this data deposit, I describe a dataset that is the result of content mining 167,318 published psychology articles for statistical test results. I tried to mine the content of HTML articles in all psychology journals published by the six major publishers in psychology, and succeeded in doing so for four major publishers (see Table 1 for descriptives per publisher). This content mining was done with the R package statcheck [1,2], which extracts statistical results from research articles in an automated fashion, given that they are reported in the format prescribed by the American Psychological Association (APA). I only inspected psychology journals, because this is a standard within the field of psychology and not necessarily outside of this field.

show abstract

“…typically .05; [Nuijten, Hartgerink, van Assen, Epskamp, & Wicherts, 2015]), H 0 is rejected and H 1 is accepted. Table 1 summarizes the four possible situations that can occur in NHST.…”

mentioning

confidence: 99%

“…This overemphasis is substantiated by the finding that more than 90% of results in the psychological literature are statistically significant (Open Science Collaboration, 2015;Sterling, Rosenbaum, & Weinkam, 1995;Sterling, 1959) despite low statistical power due to small sample sizes (Cohen, 1962;Sedlmeier, & Gigerenzer, 1989;Marszalek, Barber, Kohlhart, & Holmes, 2011;Bakker, van Dijk, & Wicherts, 2012). Consequently, publications have become biased by overrepresenting statistically significant results (Greenwald, 1975), which generally results in effect size overestimation in both individual studies (Nuijten, Hartgerink, van Assen, Epskamp, & Wicherts, 2015) and meta-analyses (van Assen, van Aert, & Wicherts, 2015;Lane, & Dunlap, 1978;Rothstein, Sutton, & Borenstein, 2005;Borenstein, Hedges, Higgins, & Rothstein, 2009). The overemphasis on statistically significant effects has been accompanied by questionable research practices (QRPs; John, Loewenstein, & Prelec, 2012) such as erroneously rounding p-values towards significance, which for example occurred for 13.8% of all p-values reported as "p = .05" in articles from eight major psychology journals in the period 1985-2013 (Hartgerink, van Aert, Nuijten, Wicherts, & van Assen, 2016).…”

mentioning

confidence: 99%

Too Good to be False: Nonsignificant Results Revisited

Hartgerink

Wicherts

Assen

2017

Collabra: Psychology

View full text Add to dashboard Cite

Due to its probabilistic nature, Null Hypothesis Significance Testing (NHST) is subject to decision errors. The concern for false positives has overshadowed the concern for false negatives in the recent debates in psychology. This might be unwarranted, since reported statistically nonsignificant findings may just be 'too good to be false'. We examined evidence for false negatives in nonsignificant results in three different ways. We adapted the Fisher test to detect the presence of at least one false negative in a set of statistically nonsignificant results. Simulations show that the adapted Fisher method generally is a powerful method to detect false negatives. We examined evidence for false negatives in the psychology literature in three applications of the adapted Fisher method. These applications indicate that (i) the observed effect size distribution of nonsignificant effects exceeds the expected distribution assuming a null-effect, and approximately two out of three (66.7%) psychology articles reporting nonsignificant results contain evidence for at least one false negative, (ii) nonsignificant results on gender effects contain evidence of true nonzero effects, and (iii) the statistically nonsignificant replications from the Reproducibility Project Psychology (RPP) do not warrant strong conclusions about the absence or presence of true zero effects underlying these nonsignificant results. We conclude that false negatives deserve more attention in the current debate on statistical practices in psychology. Potentially neglecting effects due to a lack of statistical power can lead to a waste of research resources and stifle the scientific discovery process.

show abstract

The prevalence of statistical reporting errors in psychology (1985–2013)

Cited by 427 publications

References 33 publications

When null hypothesis significance testing is unsuitable for research: a reassessment

When null hypothesis significance testing is unsuitable for research: a reassessment

688,112 Statistical Results: Content Mining Psychology Articles for Statistical Test Results

Too Good to be False: Nonsignificant Results Revisited

Contact Info

Product

Resources

About