Registered Replication Report and Extension of Maas & Hox (2005), Sufficient Sample Sizes for Multilevel Models

Abstract

The use of multilevel models in behavioral and social sciences has risen dramatically in the past twenty-five years. Researchers using these models often consult highly-referenced simulation studies when designing their own studies, especially for sample-size planning. However, current literature suggests two issues with this practice. First, recent efforts to replicate findings from seminal simulation studies suggest the rate of replicability is far from ideal. Second, with the growing popularity of multilevel models, researchers are now using more complex models; however, they often support their usage by referring to simulation studies conducted under much simpler conditions. To address these two issues, we propose a replication and extension of Maas & Hox (2005), a widely cited study that recommended Level-2 sample sizes of N = 50 to be sufficient under specific conditions. We replicate their original core findings and propose several extensions based on a literature search of multilevel modeling practices employed in fifty recently published studies. These extensions include examining conditions with imbalanced data across higher-level units, varying the number of random slopes and cross-level interactions, and varying effect sizes. Together, these results could ensure much of the existing literature relies on a replicable simulation study while also helping inform multilevel study planning for complex models which are becoming increasingly popular in behavioral and social sciences.