Evaluating significance in linear mixed-effects models in R

Abstract

Mixed-effects models are being used ever more frequently in the analysis of experimental data. However, in the lme4 package in R the standards for evaluating significance of fixed effects in these models (i.e., obtaining p-values) are somewhat vague. There are good reasons for this, but as researchers who are using these models are required in many cases to report p-values, some method for evaluating the significance of the model output is needed. This paper reports the results of simulations showing that the two most common methods for evaluating significance, using likelihood ratio tests and applying the z distribution to the Wald t values from the model output (t-as-z), are somewhat anti-conservative, especially for smaller sample sizes. Other methods for evaluating significance, including parametric bootstrapping and the Kenward-Roger and Satterthwaite approximations for degrees of freedom, were also evaluated. The results of these simulations suggest that Type 1 error rates are closest to .05 when models are fitted using REML and p-values are derived using the Kenward-Roger or Satterthwaite approximations, as these approximations both produced acceptable Type 1 error rates even for smaller samples.

The pyriform egg of the Common Murre (Uria aalge) is more stable on sloping surfaces

“…Averaged models are calculated as the conditional average, as recommended by Dormann et al (2018) when evaluating the effects of specific predictors, rather than using the model for prediction. We tested the significance of predictors in the linear mixed models using the Satterthwaite approximation (see Luke 2017).…”

Gambling on Smartphones: A Study of a Potentially Addictive Behaviour in a Naturalistic Setting

“…Significance was assessed using an ANOVA model with Sattherwaite's approximate for the degrees of freedom for each effect. This has been shown to demonstrate acceptable levels of type 1 error, even with relatively small sample sizes [36]. Cases were not analysed if they were the final gamble within a session (as PRP could not be calculated) or a substantial latency between gambles (> 60 s) was observed.…”

Is brood parasitism related to host nestling diet and nutrition?

“…''Brood size'' was always 1 for Common Cuckoo nestlings; therefore, we removed this predictor from Common Cuckoo models (Bolker 2015 Grafen and Hails 2002). P-values were computed using F-tests with Kenward-Roger corrected denominator degrees of freedom (Luke 2016); numerator degrees of freedom were always 1 (i.e. predictors were either continuous or categorical with 2 levels), except ''Diet type,'' which had numerator degrees of freedom ¼ 3.…”