Identification of a subset of patients who may be sensitive to a specific treatment is an important problem in clinical trials. In this paper, we consider the case where the treatment effect is measured by longitudinal outcomes, such as quality of life scores assessed over the duration of a clinical trial, and the subset is determined by a continuous baseline covariate, such as age and expression level of a biomarker. A threshold linear mixed model is introduced, and a smoothing maximum likelihood method is proposed to obtain the estimation of the parameters in the model. Broyden-Fletcher-Goldfarb-Shanno algorithm is employed to maximize the proposed smoothing likelihood function. The proposed procedure is evaluated through simulation studies and application to the analysis of data from a randomized clinical trial on patients with advanced colorectal cancer.

No abstract

Identification of a subset of patients who may be sensitive to a specific treatment is an important step towards personalized medicine. We consider the case where the effect of a treatment is assessed by longitudinal measurements, which may be continuous or categorical, such as quality of life scores assessed over the duration of a clinical trial. We assume that multiple baseline covariates, such as age and expression levels of genes, are available, and propose a generalized single‐index linear threshold model to identify the treatment‐sensitive subset and assess the treatment‐by‐subset interaction after combining these covariates. Because the model involves an indicator function with unknown parameters, conventional procedures are difficult to apply for inferences of the parameters in the model. We define smoothed generalized estimating equations and propose an inference procedure based on these equations with an efficient spectral algorithm to find their solutions. The proposed procedure is evaluated through simulation studies and an application to the analysis of data from a randomized clinical trial in advanced pancreatic cancer.