Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models

Abstract

In longitudinal panels and other regression models with unobserved effects, fixed effects estimation is often paired with cluster-robust variance estimation (CRVE) in order to account for heteroskedasticity and un-modeled dependence among the errors. CRVE is asymptotically consistent as the number of independent clusters increases, but can be biased downward for sample sizes often found in applied work, leading to hypothesis tests with overly liberal rejection rates. One solution is to use biasreduced linearization (BRL), which corrects the CRVE so that it is unbiased under a working model, and t-tests with Satterthwaite degrees of freedom. We propose a generalization of BRL that can be applied in models with arbitrary sets of fixed effects, where the original BRL method is undefined, and describe how to apply the method when the regression is estimated after absorbing the fixed effects. We also propose a small-sample test for multiple-parameter hypotheses, which generalizes the Satterthwaite approximation for t-tests. In simulations covering a variety of study designs, we find that conventional cluster-robust Wald tests can severely under-reject while the proposed small-sample test maintains Type I error very close to nominal levels. * The authors thank Dan Knopf for helpful discussions about the linear algebra behind the cluster-robust variance estimator.

The influence of peer behavior as a function of social and cultural closeness: A meta-analysis of normative influence on adolescent smoking initiation and continuation.

“…All the analyses were conducted in R with the robumeta package (Z. Fisher & Tipton, 2016) to perform hierarchical mixed-effects meta-regressions using the RVE approach with small-sample corrections, the clubSandwich package to perform overall tests for categorical moderators with small-sample adjustments to F-statistics in RVE (Pustejovsky & Tipton, 2016), and the meta package (Schwarzer, 2014) to implement the trim-and-fill method in the evaluation of publication bias.…”

Designing Difference in Difference Studies: Best Practices for Public Health Policy Research

“…Another pursues various forms of the cluster bootstrap (28). A third approach performs finite sample corrections based on biasreduced linearization (15,46,62,87). Cameron et al (29) provide a method for adjusting for multiway clustering; Solon et al (98) discuss the role of sampling weights.…”

Informing Progress: Insights on Personalized Learning Implementation and Effects

“…In a second change to the analysis, we now use a clustering algorithm and degrees of freedom estimators that are more robust when there are small numbers of clusters (Pustejovsky and Tipton, 2016). We cluster at the district level and we use both the treatment and VCG clusters instead of clustering on treated schools, as was done previously.…”