A Cross-Sectional Analysis of Students’ Intuitions When Interpreting CIs

Abstract

We explored how students interpret the relative likelihood of capturing a population parameter at various points of a CI in two studies. First, an online survey of 101 students found that students’ beliefs about the probability curve within a CI take a variety of shapes, and that in fixed choice tasks, 39% CI [30, 48] of students’ responses deviated from true distributions. For open ended tasks, this proportion rose to 85%, 95% CI [76, 90]. We interpret this as evidence that, for many students, intuitions about CIs distributions are ill-formed, and their responses are highly susceptible to question format. Many students also falsely believed that there is substantial change in likelihood at the upper and lower limits of the CI, resembling a cliff effect (Rosenthal and Gaito, 1963; Nelson et al., 1986). In a follow-up study, a subset of 24 post-graduate students participated in a 45-min semi-structured interview discussing the students’ responses to the survey. Analysis of interview transcripts identified several competing intuitions about CIs, and several new CI misconceptions. During the interview, we also introduced an interactive teaching program displaying a cat’s eye CI, that is, a CI that uses normal distributions to depict the correct likelihood distribution. Cat’s eye CIs were designed to help students understand likelihood distributions and the relationship between interval length, C% level and sample size. Observed changes in students’ intuitions following this teaching program suggest that a brief intervention using cat’s eyes can reduce CI misconceptions and increase accurate CI intuitions.

Perceived Statistical Knowledge Level and Self-Reported Statistical Practice Among Academic Psychologists

“…In sum, CIs have been proposed as a possible tool to help reduce NHST misconceptions. However, they are not immune to misinterpretations (Belia et al, 2005 ; Hoekstra et al, 2014 ; García-Pérez and Alcalá-Quintana, 2016 ; Kalinowski et al, 2018 ).…”

Can Visualization Alleviate Dichotomous Thinking? Effects of Visual Representations on the Cliff Effect

“…Various alternative representation styles specifically for CIs are commonly used (see, for example, [55]). In order to remedy the misunderstanding and misinterpretation of CIs, Kalinowski et al [13] designed the cat's eye confidence interval which uses normal distributions to depict the relative likelihood of values within the CI (based on the Fisherian interpretation of the CI). A violin plot with additional credible interval ranges are also used to depict arbitrary shaped (univariate) posterior distributions based on posterior samples, for example in the tidybayes R package (coined as the eye plot) [56].…”

“…Correll and Gleicher [12] studied four different visualization styles for mean and error in several settings. Kalinowski et al [13] compared students' intuitions when interpreting classic CI plots and cat's eye plots. Finally, the recent study by Hofman et al [58] focused on the impact of presenting inferential uncertainty in comparison to presenting outcome uncertainty, and investigated the effect of different visual representations of effect sizes.…”

“…Previous studies on interpretation of p-values and CIs have suggested that there are two to four confidence interpretation profiles [13], [15], [43]. While some individual variation and hybrid interpretation styles are likely to exist, due to historical reasons it is likely that the main profiles are the all-or-none category (related to Neyman-Pearson significance testing) and the gradually decreasing confidence category (related to Fisher's significance testing approach).…”

“…Despite the critique, significance testing is likely to remain a part of a scientist's toolbox. Because many of the problems with NHST are due to misunderstandings among those who conduct statistical analyses as well as among those who interpret results, work has also been conducted in making it easier to avoid common pitfalls of NHST either by altering the way analyses are conducted [7], [8], [9] or how the results are presented [10], [11], [12], [13], [14]. Instead of arguing for better methodological solutions, such as Bayesian approaches, here we study whether different styles of visual representation of common statistical problems could help to alleviate dichotomous thinking which can be approximated by studying the so-called cliff effect [15].…”