Estimation of Random Coefficient Multilevel Models in the Context of Small Numbers of Level 2 Clusters

Abstract

Multilevel data are a reality for many disciplines. Currently, although multiple options exist for the treatment of multilevel data, most disciplines strictly adhere to one method for multilevel data regardless of the specific research design circumstances. The purpose of this Monte Carlo simulation study is to compare several methods for the treatment of multilevel data specifically when there is random coefficient variation in small samples. The methods being compared are fixed effects modeling (the industry standard in business and managerial sciences), multilevel modeling using restricted maximum likelihood (REML) estimation (the industry standard in the social and behavioral sciences), multilevel modeling using the Kenward-Rogers correction, and Bayesian estimation using Markov Chain Monte Carlo. Results indicate that multilevel modeling does have an advantage over fixed effects modeling when Level 2 slope parameter variance exists. Bayesian estimation of multilevel effects can be advantageous over traditional multilevel modeling using REML, but only when prior probabilities are correctly specified. Results are presented in terms of Type I error, power, parameter estimation bias, empirical parameter estimate standard error, and parameter 95% coverage rates, and recommendations are presented.

A skills network approach to physicians’ competence in shared decision making

“…This is fairly in line with suggestions for research on clinical competence 13 and would also be sufficient for using standard maximum‐likelihood estimation methods 59 . Substantially smaller studies should preferably use Bayesian estimation with informative priors that can be derived from the results of this investigation 60 …”

“…59 Substantially smaller studies should preferably use Bayesian estimation with informative priors that can be derived from the results of this investigation. 60 In addition to testing the predictive part of the presented model, the theoretical side of our approach also needs further attention. Our hypothesis stating that the definitions of competence as a composite of proficiency in specific skills (here, using observer-rated data) and as a particular pattern of organizing behavioural skills (here, using a network model of patient-rated data) may be functionally equivalent poses an exciting topic for future work.…”

Motivation states to move, be physically active and sedentary vary like circadian rhythms and are associated with affect and arousal

“…Thus, our statistical approach is consistent with recent suggestions in the literature for analyzing multilevel data with small level two sample sizes. Similar work examining the influence of variables nested within individuals across time appears in the literature and reports similar level two sample sizes as collected in this work (e.g., (83,86,87). Still, as with any research, future work collecting larger samples is necessary to further confirm these results and extend generalizability to larger and more diverse populations.…”

“…Still the current approach is valid given that modeling level two variance (accounting for unmeasured individual differences) is particularly important, especially when the numbers of clusters is small ( 83 ). Our models also used restricted maximum likelihood (REML) for estimation– a method shown to perform well even with 10 or fewer clusters ( 83 – 85 ). Thus, our statistical approach is consistent with recent suggestions in the literature for analyzing multilevel data with small level two sample sizes.…”

Conducting and Evaluating Multilevel Studies: Recommendations, Resources, and a Checklist

“…However, although Bayesian methods improve convergence rates (Depaoli & Clifton, 2015), the use of uninformative priors does not generally overcome maximum likelihood estimates in terms of bias and power (McNeish, 2016), and it may even make them worse (McNeish, 2017a). Thus, informative priors should be chosen carefully (Bolin et al, 2019). However, informative priors do not have to be strong to be useful (McNeish, 2016).…”