Colloquium on Effect Sizes: the Roles of Editors, Textbook Authors, and the Publication Manual

Abstract

Reformers have long argued that misuse of Null Hypothesis Significance Testing (NHST) is widespread and damaging. The authors analyzed 150 articles from the Journal of Applied Psychology (JAP) covering 1940 to 1999. They examined statistical reporting practices related to misconceptions about NHST, American Psychological Association (APA) guidelines, and reform recommendations. The analysis reveals (a) inconsistency in reporting alpha and p values, (b) the use of ambiguous language in describing NHST, (c) frequent acceptance of null hypotheses without consideration of power, (d) that power estimates are rarely reported, and (e) that confidence intervals were virtually never used. APA guidelines have been followed only selectively. Research methodology reported in JAP has increased greatly in sophistication over 60 years, but inference practices have shown remarkable stability. There is little sign that decades of cogent critiques by reformers had by 1999 led to changes in statistical reporting practices in JAP.

Statistical reminders involving indirect comparisons and non-significant results: a case study

“…Finally, and most importantly, it would be remiss to perceive the aforementioned statistical issues to be specific to the paper highlighted as these are prevalent in the wider literature as well (2,6). Given this prevalence, though no systematic search has been conducted, it is not unlikely that these statistical issues exist, more specifically, in physiology as well as non-invasive brain stimulation literature.…”

Should Psychology Journals Adopt Specialized Statistical Review?

“…Because statistical power is a function of multiple factors, the problem may be less severe in domains (such as psychophysics) that commonly feature low intrasubject variability, within-subjects designs, and multiple measurement trials per subject (Rouder & Haaf, 2017). Inadequate statistical power, coupled with Bakker and Wicherts (2014); John et al 2012Incorrect calculation of effect sizes (e.g., using erroneous formulas) Hardwicke et al 2018 2012; Kerr (1998); Wagenmakers, Wetzels, Borsboom, Maas, and van der Kievit (2012) Incorrectly concluding that a nonsignificant outcome means that there is "no effect" Dienes (2014); Finch, Cumming, and Thomason (2001); Sedlmeier and Gigerenzer (1989) Assuming that the difference between significant and not significant is itself significant or analyzing interactions erroneously Nieuwenhuis, Forstmann, and Wagenmakers (2011); Gelman and Stern (2006) publication bias, can lead to inflated effect-size estimates and increases the likelihood of false negatives and false discoveries (Button et al, 2013;Fraley & Vazire, 2014;Ioannidis, 2005). Survey evidence and examination of articles' Method sections suggests that many psychologists choose sample sizes on the basis of typical practice in their domains of research rather than formal power analysis (Sedlmeier & Gigerenzer, 1989;Vankov et al, 2014).…”

Substantive significance and it´s measures

“…Je ovšem otázka, nakolik jsou tyto texty užívány pro vzdělávání sociologů, psychologů či pedagogů. Na nedostatky učebnic a výuky v zahraničí již upozornily mnohé analýzy [Kliner, Leech, Morgan 2002, Halley, Krauss 2002, Finch, Cumming, Thomason 2001, Robinson, Fouladi, Williams, Bera 2002. V České republice na kritické a analytické zhodnocení výuky a učebních textů teprve čekáme.…”