Avoiding ‘Data Snooping’ in Multilevel and Mixed Effects Models

Abstract

Summary. Multilevel or mixed effects models are commonly applied to hierarchical data. The level 2 residuals, which are otherwise known as random effects, are often of both substantive and diagnostic interest. Substantively, they are frequently used for institutional comparisons or rankings. Diagnostically, they are used to assess the model assumptions at the group level. Inference on the level 2 residuals, however, typically does not account for 'data snooping', i.e. for the harmful effects of carrying out a multitude of hypothesis tests at the same time. We provide a very general framework that encompasses both of the following inference problems: inference on the 'absolute' level 2 residuals to determine which are significantly different from 0, and inference on any prespecified number of pairwise comparisons. Thus, the user has the choice of testing the comparisons of interest. As our methods are flexible with respect to the estimation method that is invoked, the user may choose the desired estimation method accordingly. We demonstrate the methods with the London education authority data, the wafer data and the National Educational Longitudinal Study data.

Adaptive Diversity of Beech Seedlings Under Climate Change Scenarios

“…To assess the effect of provenance, population and irrigation scheme on the dependent variables (all traits used in this study), we utilized a three-level hierarchical linear modeling approach (HLM; Raudenbush and Bryk, 2002), that considers the nested structure of the data in this study. The implementation of this modeling approach is standard in a variety of disciplines (Afshartous and Wolf, 2007) with varying terminology depending on discipline (the hierarchical model is also known as the mixed-effects model, the random-coefficient model, and in the context of panel data, the repeated-measures or growth-curve model). A major advantage in this type of model over the standard regression models, is the within group and between groups comparison and the improved accuracy of point estimates in model parameters (e.g., Katahira, 2016).…”

The Limitations of using School League Tables to Inform School Choice

“…Using these overlap intervals, two schools are significantly different from each other, at the 5% level, if their overlap intervals fail to cross. We note that where parents wish to make more than one pairwise comparison, these overlap intervals should be wider (Afshartous and Wolf, 2007). Hence, the inferences we describe below give an optimistic picture of how well schools can be separated.…”

“…For making a single pairwise comparison, they show that the width of these 'overlap intervals' should, on average, be approximately 1.4 times the standard error of the school effect in order to keep the overall significance level at approximately 5%. Note that this procedure is only appropriate for parents who make just one pairwise comparison; for comparing more than two schools a multiple comparisons procedure is required (see, for example, Afshartous and Wolf, 2007).…”

Calculating and graphing within-subject confidence intervals for ANOVA

“…If multiple corrections are critical, it is best to supplement graphical presentation with formal a priori or post hoc inference using a procedure that also controls Type I error rates in a strong fashion. 18 There are also more formal treatments of the multiple comparison problem in relation to a Goldstein-Healy plot (see Afshartous & Wolf, 2007;Afshartous & Preston, 2010).…”