Beyond reporting statistical significance: Identifying informative effect sizes to improve scientific communication

Abstract

Transparent communication of research is key to foster understanding within and beyond the scientific community. An increased focus on reporting effect sizes in addition to p value–based significance statements or Bayes Factors may improve scientific communication with the general public. Across three studies ( N = 652), we compared subjective informativeness ratings for five effect sizes, Bayes Factor, and commonly used significance statements. Results showed that Cohen’s U3 was rated as most informative. For example, 440 participants (69%) found U3 more informative than Cohen’s d, while 95 (15%) found d more informative than U3, with 99 participants (16%) finding both effect sizes equally informative. This effect was not moderated by level of education. We therefore suggest that in general, Cohen’s U3 is used when scientific findings are communicated. However, the choice of the effect size may vary depending on what a researcher wants to highlight (e.g. differences or similarities).

Ensuring the comparability of cross-national survey data on intimate partner violence against women: a cross-sectional, population-based study in the European Union

“…The magnitude of these cross-national comparisons was evaluated using Cohen’s d statistic, indicating d values above 0.20, 0.50 and 0.80 for small, medium and large size effects, respectively 32. This statistic can be used to obtain the Cohen’s U 3 statistic, which indicates the percentage of cases of one country that is higher than the average of another 33–35. The Cohen’s d and U 3 statistics of the comparisons between each pair of countries can be found in the online supplementary material 1.…”

A case against default effect sizes in sport and exercise science

“…Going back to our earlier example (d z = 11.62 and 0.25, respectively), the CLES would be approximately >99% and 59.9%, or the probability of a randomly sample individual undergoing an improvement is >99% or 59.9% for intervention 1 and 2, respectively. As Hanel & Mehler (2019) demonstrated, the CLES may be a more intuitive description of the signal-to-noise SMD. While our personal recommendation leans towards the use of magnitude-based SMDs and CLES, it is up to the individual sport and exercise scientist to decide what effect size they feel is most appropriate for the data they are analyzing and point they are trying to communicate (Hönekopp, Becker & Oswald, 2006).…”

“…CLES is particularly useful because it directly conveys the direction and variability of change scores without suggesting that the mean difference itself is small or large. Further, current evidence suggests that the CLES is easier for readers to comprehend than a signal-to-noise SMD (Hanel & Mehler, 2019 1 Although conceptually similar, Glass's and pre have different distributional properties (Becker, 1988).…”

“…In our opinion, standardized effect sizes can be useful tools for interpreting data when thoughtfully employed by the scientists reporting them. However, sport and exercise scientists should be careful when selecting the appropriate SMD or effect size, and ensure that their choice effectively communicates the effect of interest (Hanel & Mehler, 2019). Herein, we will discuss things to consider when reporting an SMD, and we will close by providing general recommendations and examples that we believe sport and exercise scientists will find useful.…”

Leavers and Remainers after the Brexit referendum: More united than divided after all?

“…Statements such as ‘women are more susceptible to the side effects than men’ may be understood by the public as implying that women and men ‘constitute different groups’ (Maney, , p. 2; see also Cimpian, Brandone, & Gelman, ). Indeed, research shows that lay people overestimate differences between groups when presented textually (e.g., 'women are more anxious than men'; Hanel & Mehler, ) or graphically (e.g., barplots with restricted y ‐axes; Hanel, Maio, & Manstead, ).…”