Are Small Effects the Indispensable Foundation for a Cumulative Psychological Science? A Reply to Götz et al. (2022)

Abstract

In the January 2022 issue of Perspectives, Götz et al. argued that small effects are “the indispensable foundation for a cumulative psychological science.” They supported their argument by claiming that (a) psychology, like genetics, consists of complex phenomena explained by additive small effects; (b) psychological-research culture rewards large effects, which means small effects are being ignored; and (c) small effects become meaningful at scale and over time. We rebut these claims with three objections: First, the analogy between genetics and psychology is misleading; second, p values are the main currency for publication in psychology, meaning that any biases in the literature are (currently) caused by pressure to publish statistically significant results and not large effects; and third, claims regarding small effects as important and consequential must be supported by empirical evidence or, at least, a falsifiable line of reasoning. If accepted uncritically, we believe the arguments of Götz et al. could be used as a blanket justification for the importance of any and all “small” effects, thereby undermining best practices in effect-size interpretation. We end with guidance on evaluating effect sizes in relative, not absolute, terms.

Editorial Perspective: When is a ‘small effect’ actually large and impactful?

“…For example, G€ otz et al ( 2022) argued that small effect sizes are the norm in psychological sciences and can be highly relevant. Primbs et al (2022) make the important point that this should not be used as an argument to uncritically accept all and any small effect as important or impactful, which could be dangerous. Similarly, Anvari et al (2022) argue that it is important to consider that when an effect size is generalised to a new context, one must consider both amplifying and counteracting mechanisms rather than heuristically accepting that all effects are important.…”

Revisiting the rigidity-of-the-right hypothesis: A meta-analytic review.

“…Given that many scholars have deemed effects of these magnitudes to possess theoretical and practical importance for other notable predictors of social conservatism (e.g., low educational attainment; Lipset, 1959;Schoon et al, 2010), it seems reasonable, at first blush, to conclude that rigidity is meaningfully related to social conservatism. Moreover, even certain "small" effects can have profound real-world implications (Abelson, 1985;Funder & Ozer, 2019) and some authors posit that most psychological phenomena are an outcome of the additive influence of small effects, which may be meaningful in aggregate and/or longitudinally (Götz et al, 2022;Primbs et al, 2022; see Supplemental File 3, for an in-depth discussion of our effect size magnitudes).…”

Law and psychology must think critically about effect sizes

“…In Lewis and Wai’s terms, this makes clear what is still unknown (precise effect size estimates) and why (bias in research and reporting). Similarly, statements that effect sizes accumulate should acknowledge that there is no empirical research supporting this in many contexts [ 23 ] and provide verifiable lines of reasoning explaining why they may or may not accumulate [ 24 ]. Communication in this style gives life to more general calls for legal psychologists to take caution when communicating with nonscientists [ 11 ].…”