Comparing random effects models, ordinary least squares, or fixed effects with cluster robust standard errors for cross-classified data.

Abstract

Cross-classified random effects modeling (CCREM) is a common approach for analyzing cross-classified data in psychology, education research, and other fields. However, when the focus of a study is on the regression coefficients at level one rather than on the random effects, ordinary least squares regression with cluster robust variance estimators (OLS-CRVE) or fixed effects regression with CRVE (FE-CRVE) could be appropriate approaches. These alternative methods are potentially advantageous because they rely on weaker assumptions than those required by CCREM. We conducted a Monte Carlo Simulation study to compare the performance of CCREM, OLS-CRVE, and FE-CRVE in models, including conditions where homoscedasticity assumptions and exogeneity assumptions held and conditions where they were violated, as well as conditions with unmodeled random slopes. We found that CCREM outperformed the alternative approaches when its assumptions are all met. However, when homoscedasticity assumptions are violated, OLS-CRVE and FE-CRVE provided similar or better performance than CCREM. When the exogeneity assumption is violated, only FE-CRVE provided adequate performance. Further, OLS-CRVE and FE-CRVE provided more accurate inferences than CCREM in the presence of un-modeled random slopes. Thus, we recommend two-way FE-CRVE as a good alternative to CCREM, particularly if the homoscedasticity or exogeneity assumptions of the CCREM might be in doubt.

A practical guide to selecting and blending approaches for clustered data: Clustered errors, multilevel models, and fixed-effect models.

“…Alternatively, school affiliation dummies may be used but perhaps students are further clustered within classrooms. Clustering the errors on top of adding school affiliation dummies would result in accurate standard errors if there were dependence due to classrooms (Lee & Pustejovsky, 2023). As another example, errors from a fixed-effect model may be heteroskedastic, so clustered errors could be applied to ensure proper inference if regression assumptions beside independence are not upheld.…”

“…Hierarchies with more than one organizational unit are straightforward to accommodate with multilevel models because the variance is simply partitioned into additional sources (Goldstein, 1994; Grady & Beretvas, 2010). Models with additional levels do require more assumptions (e.g., exogeneity across more levels, distributional assumptions of more random effects) and performance can deteriorate when assumptions are not upheld (Lee & Pustejovsky, 2023).…”

“…Clustered errors are not as straightforward to apply to cross-classified data as they are with three-level nested data because Level-2a and Level-2b both need to be considered when the organizational units are not nested. Full detail will not be provided in this article (see Cameron et al, 2011 or Lee & Pustejovsky, 2023 for specific details about the computational mechanism), but the main idea is that the empirical covariance for cross-classified clustered errors is the sum of three separate calculations—clustering the errors based on Level-2a, clustering the errors based on Level-2b, and clustering the errors on the intersection of Levels-2a and 2b.…”

A Practical Guide to Selecting and Blending Approaches for Clustered Data: Clustered Errors, Multilevel Models, and Fixed Effect Models

“…Clustering the errors on top of adding school affiliation dummies would result in accurate standard errors if there were dependence due to classrooms (Lee & Pustejovsky, 2023). As another example, errors from a fixed effect model may be heteroskedastic, so clustered errors could be applied to ensure proper inference if regression assumptions beside independence are not upheld.…”

“…Hierarchies with more than one organizational unit are straightforward to accommodate with multilevel models because the variance is simply partitioned into additional sources (Goldstein, 1994;Grady & Beretvas, 2010). Models with additional levels do require more assumptions (e.g., exogeneity across more levels, distributional assumptions of more random effects) and performance can deteriorate when assumptions are not upheld (Lee & Pustejovsky, 2023). Clustered errors and fixed effects models can accommodate more than one organizational unit, often with fewer assumptions.…”