Assumption-checking rather than (just) testing: The importance of visualization and effect size in statistical diagnostics

Abstract

Statistical methods generally have assumptions (e.g., normality in linear regression models). Violations of these assumptions can cause various issues, like statistical errors and biased estimates, whose impact can range from inconsequential to critical. Accordingly, it is important to check these assumptions, but this is often done in a flawed way. Here, I first present a prevalent but problematic approach to diagnostics—testing assumptions using null hypothesis significance tests (e.g., the Shapiro–Wilk test of normality). Then, I consolidate and illustrate the issues with this approach, primarily using simulations. These issues include statistical errors (i.e., false positives, especially with large samples, and false negatives, especially with small samples), false binarity, limited descriptiveness, misinterpretation (e.g., of p-value as an effect size), and potential testing failure due to unmet test assumptions. Finally, I synthesize the implications of these issues for statistical diagnostics, and provide practical recommendations for improving such diagnostics. Key recommendations include maintaining awareness of the issues with assumption tests (while recognizing they can be useful), using appropriate combinations of diagnostic methods (including visualization and effect sizes) while recognizing their limitations, and distinguishing between testing and checking assumptions. Additional recommendations include judging assumption violations as a complex spectrum (rather than a simplistic binary), using programmatic tools that increase replicability and decrease researcher degrees of freedom, and sharing the material and rationale involved in the diagnostics.

When significant is not significant: Six clinical examples that disprove common wrong beliefs about statistical testing

“…-Once the statistical model that best fits the context has been determined, proceed with data collection and carefully evaluate -through graphs, tests, and, if necessary, sensitivity analysis -the background assumptions on which the reliability of the entire investigation depends (Mansournia et al, 2021;Shatz, 2024). The details of these examinations should be provided to the reader (e.g., in a supplementary file, see Mansournia et al, 2024).…”

A framework for public health analyses: a list of user-friendly solutions to avoid significance misconceptions in statistical testing

“…Secondly, but not less importantly, certain background assumptions (e.g., random sampling, data normality, etc.) are not only assumed true by the model but must also be ensured at the empirical-experimental level (Shatz, 2024;Rovetta, 2024 b ). Indeed, the readability and usefulness of the P-value resulting from the test are conditional on their truthfulness (Amrhein et al, 2019 b ; ).…”

“…The format shown in Table 2 (next page) can assist the researcher in this task. In accordance with Shatz (2024), graphical visualization of data properties (e.g., frequency histograms, Q-Q plots, etc.) is generally necessary to formulate appropriate conclusions.…”

Data Distribution: Normal or Abnormal?

“…Violation of this assumption may result in unreliable results and biased estimates with trivial to critical consequences. 6 7 8 9 Lack of awareness of these assumptions (e.g., normality of the data distribution) contributes to inappropriate use of statistical methods and reporting of results in scientific articles. 10 11 12 In fact, all researchers are expected to plan and report the results of their assessments of the underlying assumptions in their study protocols and manuscripts.…”