Incorporating causal factors into reinforcement learning for dynamic treatment regimes in HIV

Abstract

Background Reinforcement learning (RL) provides a promising technique to solve complex sequential decision making problems in health care domains. However, existing studies simply apply naive RL algorithms in discovering optimal treatment strategies for a targeted problem. This kind of direct applications ignores the abundant causal relationships between treatment options and the associated outcomes that are inherent in medical domains. Methods This paper investigates how to integrate causal factors into an RL process in order to facilitate the final learning performance and increase explanations of learned strategies. A causal policy gradient algorithm is proposed and evaluated in dynamic treatment regimes (DTRs) for HIV based on a simulated computational model. Results Simulations prove the effectiveness of the proposed algorithm for designing more efficient treatment protocols in HIV, and different definitions of the causal factors could have significant influence on the final learning performance, indicating the necessity of human prior knowledge on defining a suitable causal relationships for a given problem. Conclusions More efficient and robust DTRs for HIV can be derived through incorporation of causal factors between options of anti-HIV drugs and the associated treatment outcomes.

An Analysis of Reinforcement Learning for Malaria Control

“…The majority of research using RL in healthcare is in dynamic treatment regimes, where the goal is to develop effective treatment regimes that can dynamically adapt to the varying clinical states and improve the long-term outcomes for patients (Yu et al, 2019b). This includes DTR for diseases such as cancer (Zhao, Kosorok, & Zeng, 2009;Liu, Logan, Liu, Xu, Tang, & Wang, 2017), diabetes (Daskalaki, Scarnato, Diem, & Mougiakakou, 2010;Bothe, Dickens, Reichel, Tellmann, Ellger, Westphal, & Faisal, 2013;Daskalaki, Diem, & Mougiakakou, 2013), anemia (Malof & Gaweda, 2011;Escandell-Montero, Chermisi, Martinez-Martinez, Gomez-Sanchis, Barbieri, Soria-Olivas, Mari, Vila-Francés, Stopper, Gatti, et al, 2014), HIV (Parbhoo, 2014;Parbhoo, Bogojeska, Zazzi, Roth, & Doshi-Velez, 2017;Yu, Dong, Liu, & Ren, 2019a), mental illnesses (Paredes, Gilad-Bachrach, Czerwinski, Roseway, Rowan, & Hernandez, 2014;Pineau, Guez, Vincent, Panuccio, & Avoli, 2009), and DTR in critical care (Weng, Gao, He, Yan, & Szolovits, 2017;Petersen, Yang, Grathwohl, Cockrell, Santiago, An, & Faissol, 2018).…”

Reinforcement Learning in Healthcare: A Survey

“…Other approaches applied various kinds of TRL techniques so as to take advantage of the prior information from previously learned transition models [149], [150] or learned policy [151]. More recently, Yu et al [152] proposed a causal policy gradient algorithm and evaluated it in the treatment of HIV in order to facilitate the final learning performance and increase explanations of learned strategies.…”

A reinforcement learning model to inform optimal decision paths for HIV elimination

“…We use a previously validated Progression and Transmission of HIV (PATH 2.0) model [ 26 ], a dynamic stochastic agent-based model, to simulate the epidemic and evaluate the decisions. Previous RL models in HIV have focused on patient-level clinical decisions such as optimal treatment protocols [ 27 , 28 ]. Recent literature has seen an emergence in the use of RL for public health decision-making related to the COVID-19 pandemic, but they predominantly use deterministic equation-based model environments [ 29 – 34 ].…”