Bayesian methods for the analysis of small sample multilevel data with a complex variance structure.

Abstract

Inferences from multilevel models can be complicated in small samples or complex data structures. When using (restricted) maximum likelihood methods to estimate multilevel models, standard errors and degrees of freedom often need to be adjusted to ensure that inferences for fixed effects are correct. These adjustments do not address problems in estimating variance/covariance components. An alternative to the adjusted likelihood method is to use Bayesian methods, which can produce accurate inferences about fixed effects and variance/covariance parameters. In this article, the authors contrast the benefits and limitations of likelihood and Bayesian methods in the estimation of multilevel models. The issues are discussed in the context of a partially clustered intervention study, a common intervention design that has been shown to require an adjusted likelihood analysis. The authors report a Monte Carlo study that compares the performance of an adjusted restricted maximum likelihood (REML) analysis to a Bayesian analysis. The results suggest that for fixed effects, the models perform equally well with respect to bias, efficiency, and coverage of interval estimates. Bayesian models with a carefully selected gamma prior for the variance components were more biased but also more efficient with respect to estimation of the variance components than the REML model. However, the results also show that the inferences about the variance components in partially clustered studies are sensitive to the prior distribution when sample sizes are small. Finally, the authors compare the results of a Bayesian and adjusted likelihood model using data from a partially clustered intervention trial.

The Effect of Small Sample Size on Two-Level Model Estimates: A Review and Illustration

“…In total, the review included 20 studies (Austin, P.C 2010;Baldwin and Fellingham 2013;Bell, B.A et al 2014;Browne and Draper 2006;Clarke 2008;Cohen 1998;Ferron, Bell, Hess, Rendina-Gobioff, and Hibbard 2009;Hox et al 2012;Konstantopoulos 2010;Kreft 1996;Maas, C and Hox 2004;Maas and Hox 2005;McNeish 2014;Meuleman and Billiet 2009;Moineddin, Matheson, and Glazier 2007;Mok 1995;Paccagnella 2011;Scherbaum and Ferreter 2009;Snijders and Bosker 1993;Stegmueller 2013). Of these 20 studies, three focused solely on binary outcomes, 14 solely on continuous outcome, and three featured both binary and continuous outcomes.…”

Search for GeVγ-Ray Pair Halos Around Low Redshift Blazars

“…Since the number of counts in each bin (i, j) can be quite small (average counts in an individual bin N i,j ∼ 3 for the BL Lacs and N i,j ∼ 6 for the FSRQs in the 1 GeV-1.58 GeV energy range), a naive application of the MLE where one evaluates the joint likelihood L ≡ i,j P (N i,j |λ i,j ) can give large estimation errors, resulting in a non-converging distribution of the TS (the logarithmic likelihood ratio), and can potentially lead to a type II error [21]. While this is addressed by the Bayesian analysis, it may be a problematic for a frequentist inference [28]. Here we adopt a novel approach [21] where we repartition the data into two sets: the stacked angular distribution { n i=1 N i,j } ≡ {η j } obtained by summing over sources i, and the stacked source distribution { m j=1 N i,j } ≡ {ζ i } obtained by summing over angular bins j, where m and n are the total number of angular bins and stacked sources, respectively.…”

“…Different from the frequentist LRT, for a Bayesian method, the problem of limited statistics in the (i, j) bins is eliminated [28], and we can include all the information contained in the data. We are left with the straight forward (but computationally difficult) task of evaluating the multi-dimensional integral over model parameters to obtain the p-value.…”

“…To put our results in the more familiar language of frequentist statistics, we simulated the distribution of the TS by using a Monte Carlo method based on the null hypothesis. The LRT shows that if the stacked source appears to be a point source given by the Fermi PSF, the significance (probability) of getting an observation of the stacked BL Lacs in the 1GeV-1.58GeV energy bin is equivalent to the significance (probability) of getting a normal distributed sample at ∼ 2.3σ [21].Alternatively, we calculate the Bayes factors [29,30] to test the extendedemission hypothesis H 1 for given values of f halo = f * halo and Θ = Θ * (a subset of H 1 ) against the null hypothesisDifferent from the frequentist LRT, for a Bayesian method, the problem of limited statistics in the (i, j) bins is eliminated [28], and we can include all the information contained in the data. We are left with the straight forward (but computationally difficult) task of evaluating the multi-dimensional integral over model parameters to obtain the p-value.…”

Bayesian PTSD-Trajectory Analysis with Informed Priors Based on a Systematic Literature Search and Expert Elicitation

“…The reason is that the Bayesian framework can be used to "shrink" extreme estimates by incorporating information within the prior distribution (e.g., Rouder, Sun, Speckman, Lu, & Zhou, 2003). This issue has been illustrated in general (Baldwin & Fellingham, 2013), for longitudinal models , as well as with LGMMs (De la Cruz-Mesía, Quintana, & Marshall, 2008;Depaoli, 2013;Kohli, Hughes, Wang, Zopluoglu, & Davison, 2015;Lenk & Desarbo, 2000). Note that we do not mean to imply that noninformative priors always result in inaccurate model results.…”