Cluster randomised trials with a binary outcome and a small number of clusters: comparison of individual and cluster level analysis method

Abstract

Background Cluster randomised trials (CRTs) are often designed with a small number of clusters, but it is not clear which analysis methods are optimal when the outcome is binary. This simulation study aimed to determine (i) whether cluster-level analysis (CL), generalised linear mixed models (GLMM), and generalised estimating equations with sandwich variance (GEE) approaches maintain acceptable type-one error including the impact of non-normality of cluster effects and low prevalence, and if so (ii) which methods have the greatest power. We simulated CRTs with 8–30 clusters, altering the cluster-size, outcome prevalence, intracluster correlation coefficient, and cluster effect distribution. We analysed each dataset with weighted and unweighted CL; GLMM with adaptive quadrature and restricted pseudolikelihood; GEE with Kauermann-and-Carroll and Fay-and-Graubard sandwich variance using independent and exchangeable working correlation matrices. P-values were from a t-distribution with degrees of freedom (DoF) as clusters minus cluster-level parameters; GLMM pseudolikelihood also used Satterthwaite and Kenward-Roger DoF. Results Unweighted CL, GLMM pseudolikelihood, and Fay-and-Graubard GEE with independent or exchangeable working correlation matrix controlled type-one error in > 97% scenarios with clusters minus parameters DoF. Cluster-effect distribution and prevalence of outcome did not usually affect analysis method performance. GEE had the least power. With 20–30 clusters, GLMM had greater power than CL with varying cluster-size but similar power otherwise; with fewer clusters, GLMM had lower power with common cluster-size, similar power with medium variation, and greater power with large variation in cluster-size. Conclusion We recommend that CRTs with ≤ 30 clusters and a binary outcome use an unweighted CL or restricted pseudolikelihood GLMM both with DoF clusters minus cluster-level parameters.

Cluster randomized controlled trial analysis at the cluster level: The clan command

“…While the validity of the cluster-level analysis is well studied for both adjusted and unadjusted analyses (Bennett et al 2002; Ukoumunne, Carlin, and Gulliford 2007) and unadjusted analyses have been compared with individual-level analysis (Leyrat et al 2018; Thompson et al 2022), there is a need for comparisons of the adjusted clusterlevel analysis method to individual-level analysis methods to ascertain the difference in power.…”

“…However, the method is not without limitations. Unweighted cluster-level analysis can be less efficient than an individual-level analysis when cluster size varies and there are many clusters (Thompson et al 2022). Weighted cluster-level analysis using weighted least squares or a weighted t test has been proposed to improve the method efficiency, but difficulties incorporating uncertainty in the weights generally lead to standard errors that are too small and have inflated type-one errors (Westgate 2013).…”

“…The method can be used with continuous, binary, incidencerate, and ordinal outcomes. It has been found to perform well in a range of scenarios, including nonnormality of cluster-level means and with a small number of clusters (Gail et al 1996; Bennett et al 2002; Thompson et al 2022; Ukoumunne, Carlin, and Gulliford 2007).…”

“…There are several advantages to this method over individual-level methods. Clusterlevel analysis is known to maintain type-one error with as few as 4 clusters in total, whereas individual-level methods have inflated type-one errors with as many as 40 clusters and require small-sample corrections that have variable success (Leyrat et al 2018; Thompson et al 2022). Another advantage is the ease of calculating a risk ratio for a binary outcome when some individual-level methods struggle with convergence (Blizzard and Hosmer 2006).…”

Evaluating tests for cluster‐randomized trials with few clusters under generalized linear mixed models with covariate adjustment: A simulation study

“…However, when the number of clusters is small to moderate, standard tests of model coefficients may be seriously anticonservative 1 . To address inflated type I error rates due to having a small number of clusters, prior literature has proposed and validated small‐sample corrected testing procedures for linear mixed models (LMMs) and logistic GLMMs 2‐10 . The general approach of these studies is to conduct an ordinary $$$ t $$$ test or $$$ F $$$ test but with a carefully chosen denominator degree of freedom (DDF) that takes the small number of clusters into account.…”