A Blood Glucose Control Framework Based on Reinforcement Learning With Safety and Interpretability: In Silico Validation

Abstract

Controlling blood glucose levels in diabetic patients is important for managing their health and quality of life. Several algorithms based on model predictive control and reinforcement learning (RL) have been proposed so far, most of which use prior knowledge of physiological systems, the mathematical structure of blood glucose dynamics, and many episodes including failures for training the policy network in RL. To be smoothly adopted in clinical settings, we propose a fast online learning method underlining safety and interpretability. A random forest regressor and a dual attention network were exploited for glucose prediction and extension of state variables. The soft actor-critic network to determine insulin dosing was guided by proportional-integral-derivative (PID) control in the early phase, and an adaptive safe actor with suspension and additional insulin dosing was incorporated. The performance of the models was validated using an FDA-approved type 1 diabetes simulator. The results showed comparable outcomes with PID control. Using this system, glucose dynamics could be captured despite minimal prior knowledge. The extended state variables were correlated with basic states such as glucose, insulin, and meal intake, their derivatives, and their integrals, which can be fundamental elements of mathematical modeling of physiological responses. Attention scores and attribution scores in the prediction and control models represented the focused features and the internal operation of the models with interpretability. We expect this study to provide some insights on how RL can be practically adopted in clinical environments and how interpretability can provide hints of machines' thoughts for clinical applications.INDEX TERMS blood glucose control, reinforcement learning, safe and interpretable control, in silico validation, simulation for clinical application

Evaluation of Blood Glucose Level Control in Type 1 Diabetic Patients Using Deep Reinforcement Learning

“…A hybrid model-based and DRL approach is discussed by Yamagata et al 10 , which uses a discrete action space combined with meal announcement. Finally, very recently Lin et al 17 proposes a combination of machine learning methods for BG control: the controller uses a DRL SAC agent which is driven by a PID control as an initial policy and, in addition, the observation state is extended by the predictions of a dual attention network. Finally, the actions are also regulated by a adaptive safe action.…”

“…Even though different RL approaches have been increasingly proposed and discussed 5,11,12 , effective training of agents for BG control has proved to be difficult 10,11,16,17 . Several factors may explain the difficulties.…”

“…Second, compared to other learning environments, there is a remarkable number of design alternatives, whose influence is not clear and usually require careful trial and evaluation. Those involve the choice of the RL algorithm (agent) and its underlying neural network architecture, the tuning of agent hyperparameters, the selection of an appropriate reward function and even the design of the action space that may be adapted to patient specific data 11,17 . For the sake of conciseness, the design choices related to the aforementioned issues are called implementation strategy in the rest of the paper.…”

“…In this paper, we evaluate two different reinforcement learning algorithms to control in silico blood glucose levels in T1D and compare them with other well-known alternatives, including a PID controller. We describe and discuss our implementation strategy and related problems and compare it with recent proposals with a different implementation strategy 11,17 . Our paper shares with those proposals the initial premise: to train state-of-the-art Deep Reinforcement Learning (DRL) algorithms, such as SAC 20 or PPO 21 , as a T1D BG closed-loop control, but differ in several aspects of the implementation strategy that we discuss in the paper.…”

Offline Reinforcement Learning for Safer Blood Glucose Control in People with Type 1 Diabetes

“…Fox et al employed a transfer learning approach to develop dosing policies from a general patient population before fine-tuning them on target patients [13]. Lim et al used a PID controller to guide an online RL algorithm in the early stages of its learning; progressively introducing a greater proportion of RL agent actions [27]. Zhu et al developed an online RL algorithm capable of integration with a dual hormone pump, allowing control of both insulin and glucagon dosing and hence for low blood glucose corrections to be made through glucagon infusion [45].…”