Dealing with Distributional Assumptions in Preregistered Research

Abstract

Virtually any inferential statistical analysis relies on distributional assumptions of some kind. The violation of distributional assumptions can result in consequences ranging from small changes to error rates through to substantially biased estimates and parameters fundamentally losing their intended interpretations. Conventionally, researchers have conducted assumption checks after collecting data, and then changed the primary analysis technique if violations of distributional assumptions are observed. An approach to dealing with distributional assumptions that requires decisions to be made contingent on observed data is problematic, however, in preregistered research, where researchers attempt to specify all important analysis decisions prior to collecting data. Limited methodological advice is currently available regarding how to deal with the prospect of distributional assumption violations in preregistered research. In this article, we examine several strategies that researchers could use in preregistrations to reduce the potential impact of distributional assumption violations. We suggest that pre-emptively selecting analysis methods that are as robust as possible to assumption violations, performing planned robustness analyses, and/or supplementing preregistered confirmatory analyses with exploratory checks of distributional assumptions may all be useful strategies. On the other hand, we suggest that prespecifying "decision trees" for selecting data analysis methods based on the distributional characteristics of the data may not be practical in most situations.

Flexibility in reaction time analysis: many roads to a false positive?

“…In a preregistered study, all the researcher's degrees of freedom are squeezed into a predefined preprocessing and analysis pipeline [9,26,27], drawing a sharp distinction between confirmatory and exploratory analyses [28,29]. Different approaches to deal with distributional assumptions in preregistered research have been proposed [30].…”

Preregistration of secondary data analysis: A template and tutorial

“…However, note that decision trees come with their own problems and can quickly become very complex. Alternatively, you might choose to select analysis methods that make assumptions that are as conservative as possible; preregister robustness analyses which test the robustness of your findings to analysis strategies that make different assumptions; and/or pre-specify a single primary analysis strategy but note that you will also report an exploratory investigation of the validity of distributional assumptions (Williams & Albers, 2019). Of course, there are pros and cons to all methods of dealing with violations, and you should choose a technique that is most appropriate for your study.…”

Modifying food items valuation and weight with gamified executive control training

“…Any biasing of the primary statistical outcomes by ceiling or floor effect were controlled on the palatability ratings by using trimmed means (see the Data exclusion rules section), and on the palatability ratings and weight by using an additional non-parametric statistical approach (Brunner and Langer's non-parametric mixed-effects models [62,63]) in the case of non-normality of the data distributions (see [64] for a discussion).…”