Precision medicine as a control problem: Using simulation and deep reinforcement learning to discover adaptive, personalized multi-cytokine therapy for sepsis

Abstract

Sepsis is a life-threatening condition affecting one million people per year in the US in which dysregulation of the body's own immune system causes damage to its tissues, resulting in a 28 − 50% mortality rate. Clinical trials for sepsis treatment over the last 20 years have failed to produce a single currently FDA approved drug treatment. In this study, we attempt to discover an effective cytokine mediation treatment strategy for sepsis using a previously developed agent-based model that simulates the innate immune response to infection: the Innate Immune Response agent-based model (IIRABM). Previous attempts at reducing mortality with multicytokine mediation using the IIRABM have failed to reduce mortality across all patient parameterizations and motivated us to investigate whether adaptive, personalized multi-cytokine mediation can control the trajectory of sepsis and lower patient mortality. We used the IIRABM to compute a treatment policy in which systemic patient measurements are used in a feedback loop to inform future treatment. Using deep reinforcement learning, we identified a policy that achieves 0% mortality on the patient parameterization on which it was trained. More importantly, this policy also achieves 0.8% mortality over 500 randomly selected patient parameterizations with baseline mortalities ranging from 1 − 99% (with an average of 49%) spanning the entire clinically plausible parameter space of the IIRABM. These results suggest that adaptive, personalized multi-cytokine mediation therapy could be a promising approach for treating sepsis. We hope that this work motivates researchers to consider such an approach as part of future clinical trials. To the best of our knowledge, this work is the first to consider adaptive, personalized multi-cytokine mediation therapy for sepsis, and is the first to exploit deep reinforcement learning on a biological simulation.

Reinforcement Learning for Intelligent Healthcare Systems: A Comprehensive Survey

“…The obtained results in [181] depict the superior performance of the proposed mixture model compared with applying the strategies of physicians, Kernel learning only and DRL only. In [182], the authors leverage DDPG scheme to deal with the continuous state and action spaces of the sepsis environment, hence defining an effective treatment strategy for sepsis.…”

An Analysis of Reinforcement Learning for Malaria Control

“…The majority of research using RL in healthcare is in dynamic treatment regimes, where the goal is to develop effective treatment regimes that can dynamically adapt to the varying clinical states and improve the long-term outcomes for patients (Yu et al, 2019b). This includes DTR for diseases such as cancer (Zhao, Kosorok, & Zeng, 2009;Liu, Logan, Liu, Xu, Tang, & Wang, 2017), diabetes (Daskalaki, Scarnato, Diem, & Mougiakakou, 2010;Bothe, Dickens, Reichel, Tellmann, Ellger, Westphal, & Faisal, 2013;Daskalaki, Diem, & Mougiakakou, 2013), anemia (Malof & Gaweda, 2011;Escandell-Montero, Chermisi, Martinez-Martinez, Gomez-Sanchis, Barbieri, Soria-Olivas, Mari, Vila-Francés, Stopper, Gatti, et al, 2014), HIV (Parbhoo, 2014;Parbhoo, Bogojeska, Zazzi, Roth, & Doshi-Velez, 2017;Yu, Dong, Liu, & Ren, 2019a), mental illnesses (Paredes, Gilad-Bachrach, Czerwinski, Roseway, Rowan, & Hernandez, 2014;Pineau, Guez, Vincent, Panuccio, & Avoli, 2009), and DTR in critical care (Weng, Gao, He, Yan, & Szolovits, 2017;Petersen, Yang, Grathwohl, Cockrell, Santiago, An, & Faissol, 2018).…”

Reinforcement Learning in Healthcare: A Survey

“…Targeting at glycemic regulation problems for severely ill septic patients, Weng et al [204] applied PI to learn the optimal targeted blood glucose levels from real data trajectories. Petersen et al [205] investigated the cytokine mediation problem in sepsis treatment, using the DRL method, Deep Deterministic Policy Gradient (DDPG) [233], to tackle the hi-dimensional continuous states and actions, and potential-based reward shaping [234] to facilitate the learning efficiency. The proposed approach was evaluated using an agent-based model, the Innate Immune Response Agent-Based Model (IIRABM), that simulates the immune response to infection.…”