Performance of Model-Based vs. Permutation Tests in the HEALing (Helping to End Addiction Long-TermSM) Communities Study, a Covariate Constrained Cluster Randomized Trial

Abstract

Background The HEALing (Helping to End Addiction Long-termSM) Communities Study (HCS) is a multi-site parallel group cluster randomized wait-list comparison trial designed to evaluate the effect of the Communities That Heal (CTH) intervention compared to usual care on opioid overdose deaths. Covariate constrained randomization (CCR) was applied to balance the community-level baseline covariates in the HCS. The purpose of this paper is to evaluate the performance of model-based tests and permutation tests in the HCS setting. We conducted a simulation study to evaluate Type I error rates and power for model-based and permutation tests for the multi-site HCS as well as for a subgroup analysis of a single state (Massachusetts). We also investigated whether the maximum degree of imbalance in the CCR design has an impact on the performance of the tests. Methods The primary outcome, number of opioid overdose deaths, is count data assessed at the community-level that will be analyzed using a negative binomial regression model. We conducted a simulation study to evaluate the Type I error rates and power for 3 tests: 1) Wald-type t-test with small-sample corrected empirical standard error estimates; 2) Wald-type z-test with model-based standard error estimates; and 3) permutation test with test statistics calculated by the difference in average residuals for the two groups. Results Our simulation results demonstrated that Wald-type t-tests with small-sample corrected empirical standard error estimates from the negative binomial regression model maintained proper Type I error. Wald-type z-tests with model-based standard error estimates were anti-conservative. Permutation tests preserved Type I error rates if the constrained space was not too small. For all tests, power was high to detect the hypothesized 40% reduction in opioid overdose deaths for the intervention vs. comparison group both for the overall HCS and the subgroup analysis of Massachusetts (MA). Conclusions Based on the results of our simulation study, the Wald-type t-test with small-sample corrected empirical standard error estimates from a negative binomial regression model is a valid and appropriate approach for analyzing cluster-level count data from the HEALing Communities Study. Trial registration: ClinicalTrials.gov http://www.clinicaltrials.gov; Identifier: NCT04111939