Categorical data analysis: Away from ANOVAs (transformation or not) and towards logit mixed models

Abstract

This paper identifies several serious problems with the widespread use of ANOVAs for the analysis of categorical outcome variables such as forced-choice variables, question-answer accuracy, choice in production (e.g. in syntactic priming research), et cetera. I show that even after applying the arcsine-square-root transformation to proportional data, ANOVA can yield spurious results. I discuss conceptual issues underlying these problems and alternatives provided by modern statistics. Specifically, I introduce ordinary logit models (i.e. logistic regression), which are well-suited to analyze categorical data and offer many advantages over ANOVA. Unfortunately, ordinary logit models do not include random effect modeling. To address this issue, I describe mixed logit models (Generalized Linear Mixed Models for binomially distributed outcomes, Breslow & Clayton, 1993), which combine the advantages of ordinary logit models with the ability to account for random subject and item effects in one step of analysis. Throughout the paper, I use a psycholinguistic data set to compare the different statistical methods.In the psychological sciences, training in the statistical analysis of continuous outcomes (i.e. responses or independent variables) is a fundamental part of our education. The same cannot be said about categorical data analysis (Agresti, 2002; henceforth CDA), the analysis of outcomes that are either inherently categorical (e.g. the response to a yes/no question) or measured in a way that results in categorical grouping (e.g. grouping neurons into different bins based on their firing rates). CDA is common in all behavioral sciences. For example, much research on language production has investigated influences on speakers' choice between two or more possible structures (see e.g. research on syntactic persistence, Bock, 1986;Pickering and Branigan, 1998; among many others; or in research on speech errors). For language comprehension, examples of research on categorical outcomes include eye-tracking experiments (first fixations), picture identification tasks to test semantic understanding, and, of course, comprehension questions. More generally, any kind of forced-choice task, such as multiple-choice questions, and any count data constitute categorical data.Despite this preponderance of categorical data, the use of statistical analyses that have long been known to be questionable for CDA (such as analysis of variance, ANOVA) is still commonplace in our field. While there are powerful modern methods designed for CDA (e.g. ordinary and mixed logit models; see below), they are considered too complicated or simply unnecessary. There is a widely-held belief that categorical outcomes can safely be analyzed using ANOVA, if the arcsine-square-root transformation (Cochran, 1940; Rao, 1960;Winer Contact address: Assistant Professor, Brain and Cognitive Sciences, University of Rochester, Meliora Hall, Box 270268, Rochester, NY 14627-0268, P: (585) 276 3611, E: fjaeger@bcs.rochester.edu, U: http://www.bcs.rochester.ed...

The influence of verb-bound syntactic preferences on the processing of syntactic structures.

“…1 We analyzed the responses using mixed-effects logit models (Barr, Levy, Scheepers, & Tily, 2013;Jaeger, 2008;Pinheiro & Bates, 2000) in R (R Development Core Team, 2012). We started from a model including all factors and a fully specified random effect structure.…”

“…Furthermore, while speakers often choose to repeat structures across sentences (syntactic or structural priming; Bock, 1986), not all sentences influence subsequent syntactic production choices to the same extent. The strength of syntactic priming effects on production choices is inversely related to the degree to which the structure of a prime sentence was preferred (Ferreira & Bock, 2006;Scheepers, 2003), even when the syntactic preference is conditioned by the main verb (Bernolet & Hartsuiker, 2010;Jaeger & Snider, 2007, 2008Reitter, Keller, & Moore, 2011). This has been termed the inverse preference effect (Ferreira & Bock, 2006).…”

“…This hybrid model can account for inverse frequency effects in response choices: Through baselevel learning the frequent structures obtain an increased resting activation; an activation boost due to a more frequent prime structure thus leads to a smaller relative activation increase. In another prominent account, Jaeger and Snider (2007, 2008) proposed a computational model in which syntactic persistence is surprisal-sensitive. Prime structures accompanied by a larger surprisal lead to larger changes in response choices.…”

Local and neighboring patch conditions alter sex‐specific movement in banana weevils

“…Probability of movement was analyzed using GLMMs with binomial error (logit link function; Jaeger 2008) as a function of the following fixed effects: time, density of the local patch, density of the neighboring patch, sex of individuals, sex ratio of the local patch, sex ratio of the neighboring patch, bulb age, and interactions. Two crossed, random intercept effects were included: the individuals and the replicates.…”

“…Two crossed, random intercept effects were included: the individuals and the replicates. The individual‐level random intercept captures potential differences in the base probability that weevils move (Jaeger 2008). The replicate‐level random intercept captures the potential variance between replicates.…”