Personalized Dynamic Treatment Regimes in Continuous Time: A Bayesian Approach for Optimizing Clinical Decisions with Timing

Abstract

Accurate models of clinical actions and their impacts on disease progression are critical for estimating personalized optimal dynamic treatment regimes (DTRs) in medical/health research, especially in managing chronic conditions. Traditional statistical methods for DTRs usually focus on estimating the optimal treatment or dosage at each given medical intervention, but overlook the important question of "when this intervention should happen." We fill this gap by developing a two-step Bayesian approach to optimize clinical decisions with timing. In the first step, we build a generative model for a sequence of medical interventions-which are discrete events in continuous time-with a marked temporal point process (MTPP) where the mark is the assigned treatment or dosage. Then this clinical action model is embedded into a Bayesian joint framework where the other components model clinical observations including longitudinal medical measurements and time-to-event data conditional on treatment histories. In the second step, we propose a policy gradient method to learn the personalized optimal clinical decision that maximizes the patient survival by interacting the MTPP with the model on clinical observations while accounting for uncertainties in clinical observations learned from the posterior inference of the Bayesian joint model in the first step. A signature application of the proposed approach is to schedule follow-up visitations and assign a dosage at each visitation for patients after kidney transplantation. We evaluate our approach with comparison to alternative methods on both simulated and real-world datasets. In our experiments, the personalized decisions made by the proposed method are clinically useful: they are interpretable and successfully help improve patient survival.

Relative sparsity for medical decision problems

“…In our case, for example, the problem is somewhat simplified; we have discretized time, and we have not taken into account mortality and morbidity in the reward. This has allowed us to avoid methodology needed for survival analysis 71 and continuous time, 72 which could be the subject of future work. Note also that reward function learning is an active area of research 8,73 …”

Mortality prediction in ICU Using a Stacked Ensemble Model

“…[ 10 ] proposed a machine learning model based on the SOFA score for the prediction of mortality in critically ill patients. [ 38 ] developed a two-step Bayesian approach to optimize clinical decisions on timing, and the result shows that the proposed model are clinically useful to improve the survival of patients. The model of the research can be extended to other severity scoring systems, such as SOFA, GCS, and CT.…”

Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes

“…All technical proofs are provided in the Supplementary Material. In addition, in the Supplementary Material, we demonstrate the usefulness of the proposed methods by conducting a simulation study, in which we simulate a synthetic dataset that mimics a real-world electronic medical record dataset for kidney transplantation patients (Hua et al, 2021).…”

“…We demonstrate the usefulness of the proposed methods by conducting a simulation study, in which we simulate a synthetic dataset that mimics a real-world electronic medical record dataset for kidney transplantation patients (Hua et al, 2021). Kidney transplantation is the primary treatment for patients with chronic kidney disease or end-stage renal disease (Arshad et al, 2019).…”