Effect size, confidence interval and statistical significance: a practical guide for biologists

Abstract

Null hypothesis significance testing (NHST) is the dominant statistical approach in biology, although it has many, frequently unappreciated, problems. Most importantly, NHST does not provide us with two crucial pieces of information: (1) the magnitude of an effect of interest, and (2) the precision of the estimate of the magnitude of that effect. All biologists should be ultimately interested in biological importance, which may be assessed using the magnitude of an effect, but not its statistical significance. Therefore, we advocate presentation of measures of the magnitude of effects (i.e. effect size statistics) and their confidence intervals (CIs) in all biological journals. Combined use of an effect size and its CIs enables one to assess the relationships within data more effectively than the use of p values, regardless of statistical significance. In addition, routine presentation of effect sizes will encourage researchers to view their results in the context of previous research and facilitate the incorporation of results into future meta-analysis, which has been increasingly used as the standard method of quantitative review in biology. In this article, we extensively discuss two dimensionless (and thus standardised) classes of effect size statistics: d statistics (standardised mean difference) and r statistics (correlation coefficient), because these can be calculated from almost all study designs and also because their calculations are essential for meta-analysis. However, our focus on these standardised effect size statistics does not mean unstandardised effect size statistics (e.g. mean difference and regression coefficient) are less important. We provide potential solutions for four main technical problems researchers may encounter when calculating effect size and CIs: (1) when covariates exist, (2) when bias in estimating effect size is possible, (3) when data have non-normal error structure and/or variances, and (4) when data are non-independent. Although interpretations of effect sizes are often difficult, we provide some pointers to help researchers. This paper serves both as a beginner's instruction manual and a stimulus for changing statistical practice for the better in the biological sciences.

The shifting phenological landscape: Within‐ and between‐species variation in leaf emergence in a mixed‐deciduous woodland

“…All models (run using spaMM) included spatial data (X and Y coordinates) for each tree to correct for spatial autocorrelation. Support for variables was assessed based on breadth of confidence around the effect sizes; we discuss effects as statistically significant if they are more than twice the standard error around the estimate (Nakagawa & Cuthill, 2007). …”

A comparison of nanoparticulate CpG immunotherapy with and without allergens in spontaneously equine asthma‐affected horses, an animal model

“…Effect size (Cohen's d) was assessed for each piece of data to quantify the effect (mild, d  = 0.2–0.3; moderate, d  = 0.5; large, d  > 0.8). The calculation of the effect size was performed using the Effect Size Calculator8 19.…”

The fickle P value generates irreproducible results

“…This approach to statistical interpretation emphasizes the importance and precision of the estimated effect size, which answers the most frequent question that scientists ask: how big is the difference, or how strong is the relationship or association? In other words, although researchers may be conditioned to test null hypotheses (which are usually false 34 ), they really want to find not only the direction of an effect but also its size and the precision of that estimate, so that the importance and relevance of the effect can be judged 17,35,36 .…”