Estimating Multilevel Logistic Regression Models When the Number of Clusters is Low: A Comparison of Different Statistical Software Procedures

Austin, Peter C.

doi:10.2202/1557-4679.1195

Cited by 149 publications

(135 citation statements)

References 12 publications

Supporting

Mentioning

132

Contrasting

Order By: Relevance

“…For binomially distributed data (e.g., error rates), several multiple logistic regression approaches accounting for the correlation structure of the data have been proposed (Hu et al, 1998;Kuss, 2002;Lipsitz, Kim, & Zhao, 1994;Neuhaus, Kalbfleisch, & Hauck, 1991;Pendergast et al, 1996;Spiess & Hamerle, 2000). However, not much is known about how these approaches behave if small sample sizes are combined with high numbers of factor levels and with different types of population covariance matrices (e.g., Austin, 2010). Specifying the correct model can also present a challenge to researchers, due to either the high flexibility or the restrictions of the software.…”

Section: Discussionmentioning

confidence: 99%

Evaluating the robustness of repeated measures analyses: The case of small sample sizes and nonnormal data

Oberfeld

Franke

2012

Behav Res

View full text Add to dashboard Cite

Repeated measures analyses of variance are the method of choice in many studies from experimental psychology and the neurosciences. Data from these fields are often characterized by small sample sizes, high numbers of factor levels of the within-subjects factor(s), and nonnormally distributed response variables such as response times. For a design with a single within-subjects factor, we investigated Type I error control in univariate tests with corrected degrees of freedom, the multivariate approach, and a mixed-model (multilevel) approach (SAS PROC MIXED) with Kenward-Roger's adjusted degrees of freedom. We simulated multivariate normal and nonnormal distributions with varied population variancecovariance structures (spherical and nonspherical), sample sizes (N), and numbers of factor levels (K). For normally distributed data, as expected, the univariate approach with Huynh-Feldt correction controlled the Type I error rate with only very few exceptions, even if samples sizes as low as three were combined with high numbers of factor levels. The multivariate approach also controlled the Type I error rate, but it requires N ≥ K. PROC MIXED often showed acceptable control of the Type I error rate for normal data, but it also produced several liberal or conservative results. For nonnormal data, all of the procedures showed clear deviations from the nominal Type I error rate in many conditions, even for sample sizes greater than 50. Thus, none of these approaches can be considered robust if the response variable is nonnormally distributed. The results indicate that both the variance heterogeneity and covariance heterogeneity of the population covariance matrices affect the error rates.

show abstract

Section: Discussionmentioning

confidence: 99%

Evaluating the robustness of repeated measures analyses: The case of small sample sizes and nonnormal data

Oberfeld

Franke

2012

Behav Res

View full text Add to dashboard Cite

show abstract

“…Aaron (2003) reported more than 7,000 values. The accuracy of visual analyses to summarize patterns and estimate the magnitude of effects in studies like Ramsey and Ramsey (2009) has not been tested experimentally, for example, by assembling a group of methodological researchers and assessing their ability to accurately detect patterns in simulation results using artificial sets of findings varying in known ways (e.g., entirely random pattern, only one effect).…”

Section: Visual Analysis Of Simulation Resultsmentioning

confidence: 96%

“…Single-case designs in psychology and education (Kratochwill et al, 2010;Kratochwill & Levin, 1992, 2010 involve collecting and plotting repeated measures data to assess the impact of one or more interventions (Smith, 2012). A good deal of research (Bailey, 1984;DeProspero & Cohen, 1979;Jones, Vaught, & Weinrott, 1977;Knapp, 1983;Matyas & Greenwood, 1990) assessing the ability of researchers, clinicians, and others to reliably and validly detect patterns using visual analysis highlighted the difficulties of doing so even for relatively small numbers of data points (e.g., 10-15), and the use of visual and inferential analyses has been recommended (Ferron, 2002;Kratochwill et al, 2010).…”

Section: Visual Analysis Of Simulation Resultsmentioning

confidence: 99%

Experimental Design and Data Analysis in Computer Simulation Studies in the Behavioral Sciences

Harwell

Kohli

Peralta

2017

J. Mod. App. Stat. Meth.

View full text Add to dashboard Cite

show abstract

“…The performance of any particular software procedure may vary according to conditions such as the number of studies and number of observations within studies, so it is crucial to understand the strengths and limitations of each. 11,27 We used the glmer function from the R package lme4 (v. 0.999375-39), which can fit hierarchical logistic regression models, to derive empirical Bayes estimates of the study-specific effects.…”

Section: Methodsmentioning

confidence: 99%

Bayesian Hierarchical Modeling and the Integration of Heterogeneous Information on the Effectiveness of Cardiovascular Therapies

Kwok

Lewis²

2011

Circ: Cardiovascular Quality and Outcomes

View full text Add to dashboard Cite

Abstract-When making therapeutic decisions for an individual patient or formulating treatment guidelines on a population level, it is often necessary to utilize information arising from different study designs, settings, or treatments. In clinical practice, heterogeneous information is frequently synthesized qualitatively, whereas in comparative effectiveness research and guideline development, it is imperative that heterogeneous data are integrated quantitatively and in a manner that accurately captures the true uncertainty in the results. Bayesian hierarchical modeling is a technique that utilizes all available information from multiple sources and naturally yields a revised estimate of the treatment effect associated with each source. A hierarchical model consists of multiple levels (ie, a hierarchy) of probability distributions that represent relationships between information arising within single populations or trials, as well as relationships between information arising from different populations or trials. We describe the structure of Bayesian hierarchical models and discuss their advantages over simpler models when multiple information sources are relevant. Key Words: multilevel analysis Ⅲ comparative effectiveness research Ⅲ practice guidelines H ealth care providers must often decide whether to recommend a therapy for a patient for whom directly relevant treatment outcome data are limited or absent. For example, there may be limited experience with the therapy, the disease state may be rare, or the patient may be a member of a population that is underrepresented in available clinical studies. Similarly, those setting health care policy, authoring treatment guidelines, or making coverage determinations must also often make decisions regarding the effectiveness of treatments when directly relevant information is limited. An intuitive approach, frequently used by practicing clinicians, is to consider evidence from related populations and therapies, although the similarity to the patient or population of interest may vary significantly. For example, efficacy data for other drugs in the same class may be available, or the therapy may have been evaluated in populations with different comorbidities, demographic characteristics, or disease severities.Whether at the individual patient or population level, this type of decision-making requires the synthesis of information from heterogeneous sources. Many common statistical methods assume either homogeneity (eg, fixed-effects meta-analysis) or complete independence across populations and are thus ill-suited for the quantitative synthesis of heterogeneous data. Hierarchical, or multilevel, modeling is a statistical method that can be used to quantitatively and coherently combine heterogeneous information. A hierarchical model consists of multiple levels (ie, a hierarchy) of probability distributions that represent relationships between information arising within single populations or trials, as well as relationships between information arising from different popul...

show abstract

Estimating Multilevel Logistic Regression Models When the Number of Clusters is Low: A Comparison of Different Statistical Software Procedures

Cited by 149 publications

References 12 publications

Evaluating the robustness of repeated measures analyses: The case of small sample sizes and nonnormal data

Evaluating the robustness of repeated measures analyses: The case of small sample sizes and nonnormal data

Experimental Design and Data Analysis in Computer Simulation Studies in the Behavioral Sciences

Bayesian Hierarchical Modeling and the Integration of Heterogeneous Information on the Effectiveness of Cardiovascular Therapies

Contact Info

Product

Resources

About