A Primer on Practical Significance

Abstract

One of the main criticisms of NHST is that statistical significance is not practical significance. And this evaluation of the practical significance of effects often take an implicit but consequential form in the field: from informal conversations among researchers when evaluating findings, to peer reviewers deciding the importance of an article. This primer seeks to make explicit what we mean when we talk about practical significance, organize what we know of it, and assert a framework for how we can evaluate and establish it. The practical significance of effects is appraised by analyzing (i.) along different levels of analysis, (ii.) across different outcomes, (iii.) across time and (iv.) across relevant moderators; which also underlie the conditions of when small effect sizes can be consequential. Practical significance is contrasted with often conflated terms including statistical significance, effect size and effect size benchmarks as well as theoretical significance. Promising directions are then presented.

Are Small Effects the Indispensable Foundation for a Cumulative Psychological Science? A Reply to Götz et al. (2022)

“…We argue that researchers should move away from interpreting effect sizes in an absolute manner: That is, there are no small or large effects in isolation of their contextual factors. Researchers should therefore adopt a relative framework to effect-size interpretation, in which the size of an effect is compared with its costs (i.e., practical or substantive significance; Kelley & Preacher, 2012;Silan, 2020), other effects in the same empirical context (e.g., this treatment effect is larger than effect sizes of other treatments), or a benchmark such as the smallest effect size of interest or maximal positive control that is established through appropriate empirics, theory, or falsifiable justification (see Anvari & Lakens, 2021;Hilgard, 2021;Rocca & Yarkoni, 2021).…”

Teachers' proactive behaviour: Interactions with job characteristics and professional competence in a longitudinal study

“…Binnewies & Gromer, 2012; Cerit, 2017). Explaining 40% of the variance in teachers' personal initiative, improvements in the examined factors can be expected to have a practical impact on teachers ‘personal initiative (Silan, 2019). In contrast to previous studies, neither schools’ bureaucratic structures nor time pressure predicted teachers' concurrent personal initiative (cf.…”

Patterns of patient outcomes following specialist pain management in Australasia: a latent class analysis using the electronic Persistent Pain Outcomes Collaboration database

“…6 Results that are statistically significant but of less than small effect were not discussed; however, we addressed small effects where we considered they are of practical or clinical importance. 45,49 In this study, we considered small effects to be important if the variable and associated response applied to a sizable proportion of the patient population (≥25%). For example, then, within the “cause of pain” variable, the “injury at work” response was reported by 34.1% of patients; therefore, a statistically significant but small effect involving this variable and response was considered to be important.…”