A permutation procedure to detect heterogeneous treatment effects in randomized clinical trials while controlling the type I error rate

Abstract

Background/Aims Secondary analyses of randomized clinical trials often seek to identify subgroups with differential treatment effects. These discoveries can help guide individual treatment decisions based on patient characteristics and identify populations for which additional treatments are needed. Traditional analyses require researchers to pre-specify potential subgroups to reduce the risk of reporting spurious results. There is a need for methods that can detect such subgroups without a priori specification while allowing researchers to control the probability of falsely detecting heterogeneous subgroups when treatment effects are uniform across the study population. Methods We propose a permutation procedure for tuning parameter selection that allows for type I error control when testing for heterogeneous treatment effects framed within the Virtual Twins procedure for subgroup identification. We verify that the type I error rate can be controlled at the nominal rate and investigate the power for detecting heterogeneous effects when present through extensive simulation studies. We apply our method to a secondary analysis of data from a randomized trial of very low nicotine content cigarettes. Results In the absence of type I error control, the observed type I error rate for Virtual Twins was between 99% and 100%. In contrast, models tuned via the proposed permutation were able to control the type I error rate and detect heterogeneous effects when present. An application of our approach to a recently completed trial of very low nicotine content cigarettes identified several variables with potentially heterogeneous treatment effects. Conclusions The proposed permutation procedure allows researchers to engage in secondary analyses of clinical trials for treatment effect heterogeneity while maintaining the type I error rate without pre-specifying subgroups.

tehtuner: An R package to fit and tune models for the conditional average treatment effect

“…At this current time, tehtuner supports linear models fit via the LASSO (R. Tibshirani, 1996), MARS (Friedman, 1991), random forests (Breiman, 2001), and super learner (van der Laan et al, 2007) in Step 1 and linear models tuned via the LASSO (R. Tibshirani, 1996), regression and classification trees (Breiman et al, 2017), and conditional inference trees (Hothorn et al, 2006) in Step 2. Comparative evaluations of these methods can be found in Wolf et al (2022) and Deng et al (2023).…”

“…tehtuner fits models to estimate the CATE using the Virtual Twins method (Foster et al, 2011) while controlling the method's probability of falsely detecting treatment modifiers when all subjects would respond to treatment the same by implementing the permutation procedure proposed in Wolf et al (2022). A key feature of Virtual Twins is that it estimates a simple model such as a regression tree which can be easily interpreted to understand the CATE as opposed to other popular data-adaptive methods which trade in interpretability for model flexibility.…”