Correcting Fixed Effect Standard Errors When a Crossed Random Effect Was Ignored for Balanced and Unbalanced Designs

Abstract

Previous studies have detailed the consequence of ignoring a level of clustering in multilevel models with straightly hierarchical structures and have proposed methods to adjust for the fixed effect standard errors ( SEs). However, in behavioral and social science research, there are usually two or more crossed clustering levels, such as when students are cross-classified by schools and neighborhoods, yet it is not uncommon that researchers focus only on one level of clustering. Using the generalized least squares framework, in this study, we derive the bias in the fixed effect SE estimators when one crossed random effect is omitted. We then showed, using data from the Scotland Neighborhood Study, how one can correct for the SEs and obtain corrected statistical inference when a misspecified two-level model was used in a primary study, which is useful when evaluating observational studies or cluster randomized trials that ignored a crossed random effects or when conducting meta-analyses. In addition, our analytic results provide theoretical insights on how one can quantify imbalance with cross-classified data by the strength of association between the two-crossed random effects in a contingency table and how the degree of imbalance relates to the correction factor for the fixed effect SEs.

The Impact of Ignoring a Crossed Factor in Cross-Classified Multilevel Modeling

“…Generally, the SE of fixed effects estimates at the upper level is underestimated. Consistent with Lai ( 2019 ), Luo and Kwok ( 2009 , 2012 ), and Meyers and Beretvas ( 2006 ), the SE bias increased with an increased number of feeders or increased IUCC.…”

“…Several previous studies have examined the effects of the simulation factors we considered in this study. For example, Lai ( 2019 ) considered the number of clusters at each cross-classified factor, the degrees of imbalance, and cell sizes, and Ye and Daniel ( 2017 ) examined the slope of level 1 predictors, the relationships between level-2 residuals, the sample sizes of each level, and the magnitudes of intra-class correlation. However, given that all the relevant simulation conditions were not evaluated simultaneously in an integrated manner, it was difficult to understand the interaction effects among the simulation factors.…”

Propensity Score Matching with Cross-Classified Data Structures: A Comparison of Methods

“…In this study, we used the cluster-robust standard error estimator available in the survey package in R, but it is not known if this estimator is adequate for data with a cross-classified structure. Lai (2019) proposed a method for correcting standard errors of fixed-effect coefficients from two-level models to account for bias due to an omitted cross-classified factor, but the correction proposed does not account for observation weights. Because one-to-many matching with replacement requires the use of weights, it is not known if Lai's (2019) correction can be extended to this situation.…”

“…Lai (2019) proposed a method for correcting standard errors of fixed-effect coefficients from two-level models to account for bias due to an omitted cross-classified factor, but the correction proposed does not account for observation weights. Because one-to-many matching with replacement requires the use of weights, it is not known if Lai's (2019) correction can be extended to this situation.…”

Handling Missing Data in Cross-Classified Multilevel Analyses: An Evaluation of Different Multiple Imputation Approaches

“…In addition, we simulated both balanced and unbalanced samples with an average cluster size of n = 5 . This resulted in cross-classified data with a total sample size of N = 5 , 120 and moderate to strong degrees of cross-classification as indicated by Cramér’s V (i.e., in comparison with a hierarchical data structure; see Lai, 2019). 1 For the variance components, we fixed the residual variances at Level 1 ( σ 2 ) to .50 and set the variances of the random effects at Levels A, B, and AB ( τ A 2 , τ B 2 , and τ A B 2 ) to values between .10 and .30.…”