Multilevel Analysis with Few Clusters: Improving Likelihood-Based Methods to Provide Unbiased Estimates and Accurate Inference

Abstract

Quantitative comparative social scientists have long worried about the performance of multilevel models when the number of upper-level units is small. Adding to these concerns, an influential Monte Carlo study by Stegmueller (2013) suggests that standard maximum-likelihood (ML) methods yield biased point estimates and severely anti-conservative inference with few upper-level units. In this article, the authors seek to rectify this negative assessment. First, they show that ML estimators of coefficients are unbiased in linear multilevel models. The apparent bias in coefficient estimates found by Stegmueller can be attributed to Monte Carlo Error and a flaw in the design of his simulation study. Secondly, they demonstrate how inferential problems can be overcome by using restricted ML estimators for variance parameters and a t-distribution with appropriate degrees of freedom for statistical inference. Thus, accurate multilevel analysis is possible within the framework that most practitioners are familiar with, even if there are only a few upper-level units.

Do parties perceive their voter potentials correctly? Reconsidering the spatial logic of electoral competition

“…work which argues that restricted maximum-likelihood estimation as well as significance thresholds based on Satterthwaite's Degrees of Freedom approximation result in valid estimates (Elff et al 2020).…”

Changing routine activities and the decline of youth crime: A repeated cross‐sectional analysis of self‐reported delinquency in Sweden, 1999–2017*

“…Elff et al. (2021) showed in a simulation study that this method leads to unbiased estimates even with a few groups (and that estimates of group‐level variances were biased downward using standard maximum likelihood). We also compare the random intercept models with the fixed‐effects models in the “Sensitivity Analyses” section.…”

Economic inequality, income, and their effects on electoral and civil society participation in authoritarian regimes

“…However, because the country-year sample is small with around 28-54 country-years, I selected the country-year controls carefully. A recent study has shown that multilevel models with a small number of upper level-units produce unbiased estimates and confidence intervals while using frequentist maximum likelihood estimators (Elff et al 2020). Nevertheless, the results should be interpreted with caution.…”