Offline Reinforcement Learning for Safer Blood Glucose Control in People with Type 1 Diabetes

Abstract

Hybrid closed loop systems represent the future of care for people with type 1 diabetes (T1D). These devices usually utilise simple control algorithms to select the optimal insulin dose for maintaining blood glucose levels within a healthy range. Online reinforcement learning (RL) has been utilised as a method for further enhancing glucose control in these devices. Previous approaches have been shown to reduce patient risk and improve time spent in the target range when compared to classical control algorithms, but are prone to instability in the learning process, often resulting in the selection of unsafe actions. This work presents an evaluation of offline RL as a means for developing clinically effective dosing policies without the need for patient interaction. This paper examines the utility of BCQ, CQL and TD3-BC in managing the blood glucose of nine virtual patients within the UVA/Padova glucose dynamics simulator. When trained on less than a tenth of the data required by online RL approaches, this work shows that offline RL can significantly increase time in the healthy blood glucose range when compared to the strongest state-of-art baseline. This is achieved without any associated increase in low blood glucose events. Offline RL is also shown to be able to correct for common and challenging scenarios such as incorrect bolus dosing, irregular meal timings and sub-optimal training data. The code for this work is available at: https://github.com/hemerson1/offline-glucose.

A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems

“…One of the appeals of offline RL is its ability to learn a policy using previously collected data without the risk or expense of interacting with the real world. Levine et al [8] and Fu and Di [80] have extensively covered multiple real-world applications of offline RL, including robotics [9], autonomous driving [12], [81], healthcare [11], [82], dialog systems [83], and energy management systems [84].…”

“…Here, we highlight, through recent examples, a few reasons one might use offline RL over online RL in a given application. In healthcare, Emerson et al [82] used offline RL to develop a policy that selects the optimal insulin dose to maintain blood glucose levels within a healthy range. They argue that online RL is far too unstable to manage glucose levels and could cause patients to go outside of their healthy threshold.…”

Towards Safe Propofol Dosing during General Anesthesia Using Deep Offline Reinforcement Learning

“…Kondrup et al [16] used deep conservative reinforcement learning to determine the best ventilator settings for ICU patients. Emerson et al [17] proposed using ORL to learn a safer blood glucose control strategy for people with Type 1 diabetes. Shiranthika et al [18] developed the supervised optimal chemotherapy regimen, which can provide cancer patients with an optimal chemotherapy-dosing schedule, thus assisting oncologists in clinical decision-making.…”