Inverse reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units

Abstract

Background Reinforcement learning (RL) provides a promising technique to solve complex sequential decision making problems in health care domains. To ensure such applications, an explicit reward function encoding domain knowledge should be specified beforehand to indicate the goal of tasks. However, there is usually no explicit information regarding the reward function in medical records. It is then necessary to consider an approach whereby the reward function can be learned from a set of presumably optimal treatment trajectories using retrospective real medical data. This paper applies inverse RL in inferring the reward functions that clinicians have in mind during their decisions on weaning of mechanical ventilation and sedative dosing in Intensive Care Units (ICUs). Methods We model the decision making problem as a Markov Decision Process, and use a batch RL method, Fitted Q Iterations with Gradient Boosting Decision Tree , to learn a suitable ventilator weaning policy from real trajectories in retrospective ICU data. A Bayesian inverse RL method is then applied to infer the latent reward functions in terms of weights in trading off various aspects of evaluation criterion. We then evaluate how the policy learned using the Bayesian inverse RL method matches the policy given by clinicians, as compared to other policies learned with fixed reward functions. Results Results show that the inverse RL method is capable of extracting meaningful indicators for recommending extubation readiness and sedative dosage, indicating that clinicians pay more attention to patients’ physiological stability (e.g., heart rate and respiration rate), rather than oxygenation criteria ( FiO 2 , PEEP and SpO 2 ) which is supported by previous RL methods. Moreover, by discovering the optimal weights, new effective treatment protocols can be suggested. Conclusions Inverse RL is an effective approach to discovering clinicians’ underlying reward functions for designing better treatment protocols in the ventilation weaning and sedative dosing in future ICUs.

Supervised-actor-critic reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units

“…Nemati et al [18] used deep RL methods to calculate optimal unfractionated Heparin from sub-optimal clinical ICU data. Yu et al [19] used inverse RL to infer the reward functions when dealing with mechanical ventilation and sedative dosing in ICUs. Chang et al [20] proposed a Q-learning method that jointly minimized the measurement cost and maximized predictive gain, by scheduling strategically-timed measurements in ICUs.…”

“…After preprocessing, we take 10 minutes as the frequency of time series from admission time to discharge time. Please refer to [19] for more details in data processing.…”

“…Reward: The reward function r t+1 is defined as r t+1 = r vitals t+1 + r vent off t+1 + r vent on t+1 [16,19], in which r vitals t+1 evaluates the effect of these actions on the physiological stability of the patient within a reasonable range, r vent off t+1 estimates the performance of ventilation being stopped at time t +1, and r vent on t+1 simply represents the cost per hour on the ventilator.…”

Improving the informatics competency of critical care nurses: results of an interventional study in the southeast of Iran

“…Patients in critical settings need sophisticated high-tech equipment, and specialized clinicians continuously monitor them. The patient safety and patient outcomes are more vital in such settings than in other units [13,[17][18][19]. The literature review has recognized the gap in NI competency required for nurses in different work environments and critical care settings.…”

Improving the informatics competency of critical care nurses: results of an interventional study in the southeast of Iran

“…Patients in critical settings need sophisticated hightech equipment, and specialized clinicians continuously monitor them. The patient safety and patient outcomes are more vital in such settings than in other units [13,[17][18][19]. The literature review has recognized the gap in NI competency required for nurses in different work environments and critical care settings.…”