Probability as certainty: Dichotomous thinking and the misuse ofp values

Hoekstra, Rink; Finch, Sue; Kiers, Henk A.L.; Johnson, Addie

doi:10.3758/bf03213921

Cited by 97 publications

(105 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The sacred α = .05 criterion amplifies this illusion, since results end up being either "significant" or "non-significant". In a researcher's mind, significant results have passed the rigorous test of statistics and are declared "valid" -uncertainty almost ceases to exist, and sample means often end up being interpreted as being exact (Vicente and Torenvliet, 2000, pp.252-258;Hoekstra et al, 2006). For example, this amounts to saying that in Figure 4a, each bar with p < .05 should be trusted fully.…”

Section: Misjudgments Of Uncertaintymentioning

confidence: 99%

Fair Statistical Communication in HCI

Dragicevic

2016

Human–Computer Interaction Series

270

247

View full text Add to dashboard Cite

Statistics are tools to help end users accomplish their task. In research, to be qualified as usable, statistical tools should help researchers advance scientific knowledge by supporting and promoting the effective communication of research findings. Yet areas such as human-computer interaction (HCI) have adopted toolsi.e., p-values and dichotomous testing procedures -that have proven to be poor at supporting these tasks. The abusive use of these procedures has been severely criticized in a range of disciplines for several decades, suggesting that tools should be blamed, not end users. This chapter explains in a non-technical manner why it would be beneficial for HCI to switch to an estimation approach, i.e., reporting informative charts with effect sizes and interval estimates, and offering nuanced interpretations of our results. Advice is offered on how to communicate our empirical results in a clear, accurate, and transparent way without using any tests or p-values.

show abstract

Section: Misjudgments Of Uncertaintymentioning

confidence: 99%

Fair Statistical Communication in HCI

Dragicevic

2016

Human–Computer Interaction Series

270

247

View full text Add to dashboard Cite

show abstract

“…Performing a significance test and retaining α = .05 seems the default analysis strategy in psychology; it has even been called a "mindless" "null ritual" (Gigerenzer, 2004;Gigerenzer, Krauss, & Vitouch, 2004). However, there are many misunderstandings surrounding the meaning of (non)significant p-values, and their evidential value is often overestimated (Hoekstra, Finch, Kiers, & Johnson, 2006;Tversky & Kahneman, 1971).…”

Section: Understanding Statisticsmentioning

confidence: 99%

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

Nuijten¹

2018

Preprint

View full text Add to dashboard Cite

SUMMARY DOCTORAL DISSERTATION: Psychology is facing a “replication crisis”. Many psychological findings could not be replicated in novel samples, which lead to the growing concern that many published findings are overly optimistic or even false. In this dissertation, we investigated potential indicators of problems in the published psychological literature. In Part I of this dissertation, we looked at inconsistencies in reported statistical results in published psychology papers. To facilitate our research, we developed the free tool statcheck; a “spellchecker” for statistics. In Part II, we investigated bias in published effect sizes. We showed that in the presence of publication bias, the overestimation of effects can become worse if you combine studies. Indeed, in meta-analyses from the social sciences we found strong evidence that published effects are overestimated. These are worrying findings, and it is important to think about concrete solutions to improve the quality of psychological research. Some of the solutions we propose are preregistration, replication, and transparency. We argue that to select the best strategies to improve psychological science, we need research on research: meta-research.

show abstract

“…The most serious mistake relevant to our paper is that many researchers accept the null-hypothesis and claim no effect in case of a statistically nonsignificant effect (about 60%, see Hoekstra, Finch, Kiers, & Johnson, 2016). Hence, most researchers overlook that the outcome of hypothesis testing is probabilistic (if the null-hypothesis is true, or the alternative hypothesis is true and power is less than 1) and interpret outcomes of hypothesis testing as reflecting the absolute truth.…”

mentioning

confidence: 87%

“…Our results in combination with results of previous studies suggest that publication bias mainly operates on results of tests of main hypotheses, and less so on peripheral results. Another venue for future research is using the Fisher test to re-examine evidence in the literature on certain other effects or often-used covariates, such as age and race, or to see if it helps researchers prevent dichotomous thinking with individual p-values (Hoekstra, Finch, Kiers, & Johnson, 2016).…”

Section: Limitations and Further Researchmentioning

confidence: 99%

Too Good to be False: Nonsignificant Results Revisited

Hartgerink

Wicherts

Assen

2017

Collabra: Psychology

View full text Add to dashboard Cite

Due to its probabilistic nature, Null Hypothesis Significance Testing (NHST) is subject to decision errors. The concern for false positives has overshadowed the concern for false negatives in the recent debates in psychology. This might be unwarranted, since reported statistically nonsignificant findings may just be 'too good to be false'. We examined evidence for false negatives in nonsignificant results in three different ways. We adapted the Fisher test to detect the presence of at least one false negative in a set of statistically nonsignificant results. Simulations show that the adapted Fisher method generally is a powerful method to detect false negatives. We examined evidence for false negatives in the psychology literature in three applications of the adapted Fisher method. These applications indicate that (i) the observed effect size distribution of nonsignificant effects exceeds the expected distribution assuming a null-effect, and approximately two out of three (66.7%) psychology articles reporting nonsignificant results contain evidence for at least one false negative, (ii) nonsignificant results on gender effects contain evidence of true nonzero effects, and (iii) the statistically nonsignificant replications from the Reproducibility Project Psychology (RPP) do not warrant strong conclusions about the absence or presence of true zero effects underlying these nonsignificant results. We conclude that false negatives deserve more attention in the current debate on statistical practices in psychology. Potentially neglecting effects due to a lack of statistical power can lead to a waste of research resources and stifle the scientific discovery process.

show abstract

Probability as certainty: Dichotomous thinking and the misuse ofp values

Cited by 97 publications

References 21 publications

Fair Statistical Communication in HCI

Fair Statistical Communication in HCI

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

Too Good to be False: Nonsignificant Results Revisited

Contact Info

Product

Resources

About