Preparing for the next COVID: Deep Reinforcement Learning trained Artificial Intelligence discovery of multi-modal immunomodulatory control of systemic inflammation in the absence of effective anti-microbials

Abstract

Background: Despite a great deal of interest in the application of artificial intelligence (AI) to sepsis/critical illness, most current approaches are limited in their potential impact: prediction models do not (and cannot) address the lack of effective therapeutics and current approaches to enhancing the treatment of sepsis focus on optimizing the application of existing interventions, and thus cannot address the development of new treatment options/modalities. The inability to test new therapeutic applications was highlighted by the generally unsatisfactory results from drug repurposing efforts in COVID-19. Hypothesis: Addressing this challenge requires the application of simulation-based, model-free deep reinforcement learning (DRL) in a fashion akin to training the game-playing AIs. We have previously demonstrated the potential of this method in the context of bacterial sepsis in which the microbial infection is responsive to antibiotic therapy. The current work addresses the control problem of multi-modal, adaptive immunomodulation in the circumstance where there is no effective anti-pathogen therapy (e.g., in a novel viral pandemic or in the face of resistant microbes). Methods: This is a proof-of-concept study that determines the controllability of sepsis without the ability to pharmacologically suppress the pathogen. We use as a surrogate system a previously validated agent-based model, the Innate Immune Response Agent-based Model (IIRABM), for control discovery using DRL. The DRL algorithm trains an AI on simulations of infection where both the control and observation spaces are limited to operating upon the defined immune mediators included in the IIRABM (a total of 11). Policies were learned using the Deep Deterministic Policy Gradient approach, with the objective function being a return to baseline system health. Results: DRL trained an AI policy that improved system mortality from 85% to 10.4%. Control actions affected every one of the 11 targetable cytokines and could be divided into those with static/unchanging controls and those with variable/adaptive controls. Adaptive controls primarily targeted 3 different aspects of the immune response: 2nd order pro-inflammation governing TH1/TH2 balance, primary anti-inflammation, and inflammatory cell proliferation. Discussion: The current treatment of sepsis is hampered by limitations in therapeutic options able to affect the biology of sepsis. This is heightened in circumstances where no effective antimicrobials exist, as was the case for COVID-19. Current AI methods are intrinsically unable to address this problem; doing so requires training AIs in contexts that fully represent the counterfactual space of potential treatments. The synthetic data needed for this task is only possible through the use of high-resolution, mechanism-based simulations. Finally, being able to treat sepsis will require a reorientation as to the sensing and actuating requirements needed to develop these simulations and bring them to the bedside

Preparing for the next Pandemic: Simulation-based Deep Reinforcement Learning to discover and test multimodal control of systemic inflammation using repurposed immunomodulatory agents

“…The ability to manipulate any combination of mediators present is meant to simulate the potential use of combinations of interventions, which our prior work has suggested is necessary to effectively control sepsis (15)(16)(17); the DRL approach is intended to assist in addressing the exponential combinatorial issues associated with multi-drug therapy and the additional challenge needing to modify a particular treatment application to account for the temporal heterogeneity among individuals with regards to their disease trajectories.…”

“…We have previously reported on the challenges present in attempting to control sepsis using anti-cytokine/anti-mediator therapies, primarily stemming from the failures to recognize the dynamic complexity of the mechanistic processes ostensibly being targeted (12) and that in order to be effective the treatment of sepsis should be considered a complex control problem (13). In previous work we have shown that sepsis is potentially controllable by discovering multi-modal control strategies using different types of machine learning (ML) methods trained on a complex agent-based model of acute systemic inflammation (the Innate Immune Response Agent-based Model, or IIRABM ( 14)) (15)(16)(17). Specifically, the latter projects described in Refs (16,17) utilized the method, Deep Reinforcement Learning (DRL), employed by ML/Artificial Intelligence (AI) systems to successfully play and win a series of games against human experts (18)(19)(20)(21)(22).…”

“…In previous work we have shown that sepsis is potentially controllable by discovering multi-modal control strategies using different types of machine learning (ML) methods trained on a complex agent-based model of acute systemic inflammation (the Innate Immune Response Agent-based Model, or IIRABM ( 14)) (15)(16)(17). Specifically, the latter projects described in Refs (16,17) utilized the method, Deep Reinforcement Learning (DRL), employed by ML/Artificial Intelligence (AI) systems to successfully play and win a series of games against human experts (18)(19)(20)(21)(22). We term this approach simulation-based model-free DRL, and in prior work applied to method where we treated the attempt to control sepsis as a "game" to be played using the IIRABM, where potential cytokine interventions represented the "moves" implemented by the AI agent (16,17).…”

“…Specifically, the latter projects described in Refs (16,17) utilized the method, Deep Reinforcement Learning (DRL), employed by ML/Artificial Intelligence (AI) systems to successfully play and win a series of games against human experts (18)(19)(20)(21)(22). We term this approach simulation-based model-free DRL, and in prior work applied to method where we treated the attempt to control sepsis as a "game" to be played using the IIRABM, where potential cytokine interventions represented the "moves" implemented by the AI agent (16,17).…”

“…We propose that simulation-based control discovery using DRL can provide useful insights and potentially critical capabilities in designing effective multi-modal and adaptive immunomodulatory therapies for infections for which no effective anti-microbial agents exist. We have previously demonstrated in a proof-of-concept report that such a control policy can be discovered with DRL when manipulating up to 11 different mediators and soluble factors every 6 minutes (17). We now extend that study to evaluate whether DRL can train an artificial neural network (ANN) to discover a treatment policy utilizing existing anti-cytokine drugs to improve the outcomes to simulated infection in the absence of anti-microbial treatment.…”