Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies

Abstract

Many psychologists do not realize that exploratory use of the popular multiway analysis of variance harbors a multiple-comparison problem. In the case of two factors, three separate null hypotheses are subject to test (i.e., two main effects and one interaction). Consequently, the probability of at least one Type I error (if all null hypotheses are true) is 14 % rather than 5 %, if the three tests are independent. We explain the multiple-comparison problem and demonstrate that researchers almost never correct for it. To mitigate the problem, we describe four remedies: the omnibus F test, control of the familywise error rate, control of the false discovery rate, and preregistration of the hypotheses.

Phonological and orthographic influences in the bouba–kiki effect

“…All other interactions were insignificant (all F's \ 1, all p's [ 0.05), including any three-way interaction between all factors. (Cramer et al, 2014), given both the use of multiple post hoc ANOVAs to explore effects of stop/continuant status and the general use of multi-way ANOVAs. 3 Due to the fully crossed nature of our items, constrained by facts of English phonology and orthography described in the materials section, we also made post hoc analyses using two additional ANOVAs: one where stop/continuant status was included in lieu of voicing (shape 9 orthography 9 stop/continuant) and one in which stop/continuant status was included in lieu of orthography (shape 9 voicing 9 stop/continuant).…”

Do p Values Lose Their Meaning in Exploratory Analyses? It Depends How You Define the Familywise Error Rate

“…The first problem is that it leads to confusion over which hypotheses to include in the family of hypotheses that is used to compute an adjusted alpha level (Feise, 2002;O'Keefe, 2003O'Keefe, , 2007Trafimow & Earp, 2017). For example, a family could include all of the hypotheses in a multiway analysis of variance (ANOVA; Cramer et al, 2016), all of the hypotheses in a single study or multistudy article (sometimes called the experimentwise error rate), all of the hypotheses in a collection of articles that address the same issue, or even all of the hypotheses that have been and/or will be conducted by a specific researcher during their career (Hurlbert & Lombardi, 2012;O'Keefe, 2003O'Keefe, , 2007Trafimow & Earp, 2017). The argument that p values lose their meaning in exploratory analyses is based on the assumption that all of the hypotheses in a study or multistudy article should be included in a family.…”

“…However, the decision to limit the definition of a family of hypotheses to those in a study or article is an arbitrary one (O'Keefe, 2003(O'Keefe, , 2007, and other arbitrary decisions lead to different conclusions regarding the meaningfulness of p values. For example, if the familywise error rate is based on hypotheses that are tested in a specific analysis (e.g., a multiway ANOVA; Cramer et al, 2016), then the number of hypotheses that are tested in that analysis is limited and knowable, and alpha level adjustment becomes possible, even in exploratory analyses. Alternatively, if the family of hypotheses is broadened to include all of the different hypotheses that are undertaken by a specific researcher (O'Keefe, 2003(O'Keefe, , 2007Trafimow & Earp, 2017), then even p values in preregistered confirmatory analyses lose their meaning due to an unknown inflation of the alpha level based on future confirmatory tests that are undertaken by that researcher (Frane, 2015).…”

Manipulating the reported age in earliest memories

“…Group differences were examined using analyses of variance (ANOVAs), t-tests and, if assumptions were not met, nonparametric Mann-Whitney U tests. We used a sequential Bonferroni-Holm procedure in the exploratory ANOVAs to control for family-wise error rates (Cramer et al, 2016).…”