Interpretation of significance levels by psychological researchers: The .05 cliff effect may be overstated

Poitevineau, Jacques; Lecoutre, Bruno

doi:10.3758/bf03196227

Cited by 32 publications

(34 citation statements)

References 16 publications

(17 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, for the comparisons made during this study, univariate descriptive tests, the Kruskal-Wallis H test, and the Mann-Whitney test were used. The significance values of the tests followed the classical value in the literature (α = 0.05) [62].…”

Section: Data Analysesmentioning

confidence: 99%

Evaluating Activation and Absence of Negative Effect: Gamification and Escape Rooms for Learning

Belmonte

Robles

Cabrera

et al. 2020

IJERPH

View full text Add to dashboard Cite

Innovation has allowed for and developed new ways of teaching and learning. Gamification is among the new training methodologies, which is a didactic approach based on the game structure with an attractive component for students. Within gamification, flipped learning and problem-based learning, escape rooms can be found as a technical aspect, which is focused on providing enigmas and tracks for the various educational content that students have assimilated through learning based on problem solving. The aim of this study is to identify how the use of gamification with the use of educational escape rooms affects activation and absence of a negative effect on students. 61 Master students of the Autonomous City of Ceuta participated in this case study. They were divided into three study groups (1 control group; 2 experimental groups) that followed different formative actions (control group—traditional; experimental groups—escape rooms). To achieve the objectives, a mixed research design based on quantitative and qualitative techniques was followed. The instrument used for data collection was the GAMEX (Gameful Experience Scale). The results reveal that the students who had taken a gamified formative action through escape rooms obtained better assessment results in the indicators concerning motivation, teamwork, commitment, activation, and absence of a negative effect on the learning process than those with the traditional methodology.

show abstract

Section: Data Analysesmentioning

confidence: 99%

Evaluating Activation and Absence of Negative Effect: Gamification and Escape Rooms for Learning

Belmonte

Robles

Cabrera

et al. 2020

IJERPH

View full text Add to dashboard Cite

show abstract

“…When asked about the degree of confidence that an experimental treatment really has an effect, the average researcher’s confidence was found to drop quite sharply as p -values rose to about 0.1, but then confidence levelled off until 0.9, essentially showing a graded response (Poitevineau & Lecoutre, 2001). Nuzzo (2015) cites Matthew Hankins, who has collected “more than 500 creative phrases that researchers use to convince readers that their nonsignificant results are worthy of attention (see go.nature.com/pwctoq).…”

Section: Introductionmentioning

confidence: 99%

The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research

Amrhein

Korner‐Nievergelt

Roth

2017

PeerJ

266

216

View full text Add to dashboard Cite

The widespread use of ‘statistical significance’ as a license for making a claim of a scientific finding leads to considerable distortion of the scientific process (according to the American Statistical Association). We review why degrading p-values into ‘significant’ and ‘nonsignificant’ contributes to making studies irreproducible, or to making them seem irreproducible. A major problem is that we tend to take small p-values at face value, but mistrust results with larger p-values. In either case, p-values tell little about reliability of research, because they are hardly replicable even if an alternative hypothesis is true. Also significance (p ≤ 0.05) is hardly replicable: at a good statistical power of 80%, two studies will be ‘conflicting’, meaning that one is significant and the other is not, in one third of the cases if there is a true effect. A replication can therefore not be interpreted as having failed only because it is nonsignificant. Many apparent replication failures may thus reflect faulty judgment based on significance thresholds rather than a crisis of unreplicable research. Reliable conclusions on replicability and practical importance of a finding can only be drawn using cumulative evidence from multiple independent studies. However, applying significance thresholds makes cumulative knowledge unreliable. One reason is that with anything but ideal statistical power, significant effect sizes will be biased upwards. Interpreting inflated significant results while ignoring nonsignificant results will thus lead to wrong conclusions. But current incentives to hunt for significance lead to selective reporting and to publication bias against nonsignificant findings. Data dredging, p-hacking, and publication bias should be addressed by removing fixed significance thresholds. Consistent with the recommendations of the late Ronald Fisher, p-values should be interpreted as graded measures of the strength of evidence against the null hypothesis. Also larger p-values offer some evidence against the null hypothesis, and they cannot be interpreted as supporting the null hypothesis, falsely concluding that ‘there is no effect’. Information on possible true effect sizes that are compatible with the data must be obtained from the point estimate, e.g., from a sample average, and from the interval estimate, such as a confidence interval. We review how confusion about interpretation of larger p-values can be traced back to historical disputes among the founders of modern statistics. We further discuss potential arguments against removing significance thresholds, for example that decision rules should rather be more stringent, that sample sizes could decrease, or that p-values should better be completely abandoned. We conclude that whatever method of statistical inference we use, dichotomous threshold thinking must give way to non-automated informed judgment.

show abstract

“…The shapes selected for Task 2 were prompted partly by considering Neyman–Pearson, Fisherian and hybrid approaches to hypothesis testing, and the patterns found by Poitevineau and Lecoutre (2001). In a sense, responses to Task 2 present a lower bound for the prevalence of each SLD.…”

Section: Methodsmentioning

confidence: 99%

“…Poitevineau and Lecoutre (2001) asked researchers to rate their confidence that an effect existed at 12 different p -values ranging from 0.9 to 0.001 ( Figure 4 ). They identified three categories of response: An all-or-none curve (very high degree of confidence when p < 0.05 and almost no confidence otherwise), a negative linear curve and a decreasing exponential.…”

Section: Introductionmentioning

confidence: 99%

A Cross-Sectional Analysis of Students’ Intuitions When Interpreting CIs

2018

View full text Add to dashboard Cite

We explored how students interpret the relative likelihood of capturing a population parameter at various points of a CI in two studies. First, an online survey of 101 students found that students’ beliefs about the probability curve within a CI take a variety of shapes, and that in fixed choice tasks, 39% CI [30, 48] of students’ responses deviated from true distributions. For open ended tasks, this proportion rose to 85%, 95% CI [76, 90]. We interpret this as evidence that, for many students, intuitions about CIs distributions are ill-formed, and their responses are highly susceptible to question format. Many students also falsely believed that there is substantial change in likelihood at the upper and lower limits of the CI, resembling a cliff effect (Rosenthal and Gaito, 1963; Nelson et al., 1986). In a follow-up study, a subset of 24 post-graduate students participated in a 45-min semi-structured interview discussing the students’ responses to the survey. Analysis of interview transcripts identified several competing intuitions about CIs, and several new CI misconceptions. During the interview, we also introduced an interactive teaching program displaying a cat’s eye CI, that is, a CI that uses normal distributions to depict the correct likelihood distribution. Cat’s eye CIs were designed to help students understand likelihood distributions and the relationship between interval length, C% level and sample size. Observed changes in students’ intuitions following this teaching program suggest that a brief intervention using cat’s eyes can reduce CI misconceptions and increase accurate CI intuitions.

show abstract

Interpretation of significance levels by psychological researchers: The .05 cliff effect may be overstated

Cited by 32 publications

References 16 publications

Evaluating Activation and Absence of Negative Effect: Gamification and Escape Rooms for Learning

Evaluating Activation and Absence of Negative Effect: Gamification and Escape Rooms for Learning

The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research

A Cross-Sectional Analysis of Students’ Intuitions When Interpreting CIs

Contact Info

Product

Resources

About