Control of Blood Glucose for Type-1 Diabetes by Using Reinforcement Learning with Feedforward Algorithm

Ngo, Phuong D.; Wei, Susan; Holubová, Anna; Mužík, Jan; Godtliebsen, Fred

doi:10.1155/2018/4091497

Cited by 25 publications

(23 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…It is also common to find some reference values related to normal, hyper and hypoglycemia ranges in order to establish good rewards and penalties. However, we found that only five papers include the actions taken in the reward function [31,32,38,42,44]. We think it could be interesting to also consider the insulin doses in the reward function, which for example can lead to take less aggressive actions for the patients.…”

Section: Discussionmentioning

confidence: 99%

Reinforcement learning application in diabetes blood glucose control: A systematic review

Tejedor

Woldaregay

Godtliebsen

2020

Artificial Intelligence in Medicine

Self Cite

119

View full text Add to dashboard Cite

Background: Reinforcement learning (RL) is a computational approach to understanding and automating goal-directed learning and decision-making. It is designed for problems which include a learning agent interacting with its environment to achieve a goal. For example, blood glucose (BG) control in diabetes mellitus (DM), where the learning agent and its environment are the controller and the body of the patient respectively. RL algorithms could be used to design a fully closed-loop controller, providing a truly personalized insulin dosage regimen based exclusively on the patient's own data. Objective:In this review we aim to evaluate state-of-the-art RL approaches to designing BG control algorithms in DM patients, reporting successfully implemented RL algorithms in closedloop, insulin infusion, decision support and personalized feedback in the context of DM.Methods: An exhaustive literature search was performed using different online databases, analyzing the literature from 1990 to 2019. In a first stage, a set of selection criteria were established in order to select the most relevant papers according to the title, keywords and abstract. Research questions were established and answered in a second stage, using the information extracted from the articles selected during the preliminary selection.Results: The initial search using title, keywords, and abstracts resulted in a total of 404 articles.After removal of duplicates from the record, 347 articles remained. An independent analysis and screening of the records against our inclusion and exclusion criteria defined in Methods section resulted in removal of 296 articles, leaving 51 relevant articles. A full-text assessment was conducted on the remaining relevant articles, which resulted in 29 relevant articles that were critically analyzed. The inter-rater agreement was measured using Cohen Kappa test, and disagreements were resolved through discussion. Conclusions:The advances in health technologies and mobile devices have facilitated the implementation of RL algorithms for optimal glycemic regulation in diabetes. However, there exists few articles in the literature focused on the application of these algorithms to the BG regulation problem. Moreover, such algorithms are designed for control tasks as BG adjustment and their use have increased recently in the diabetes research area, therefore we foresee RL algorithms will be used more frequently for BG control in the coming years. Furthermore, in the literature there is a lack of focus on aspects that influence BG level such as meal intakes and physical activity (PA), which should be included in the control problem. Finally, there exists a need to perform clinical validation of the algorithms.

show abstract

Section: Discussionmentioning

confidence: 99%

Reinforcement learning application in diabetes blood glucose control: A systematic review

Tejedor

Woldaregay

Godtliebsen

2020

Artificial Intelligence in Medicine

Self Cite

119

View full text Add to dashboard Cite

show abstract

“…Our proposed algorithm provides more precise insulin dosage recommendation considering the patient’s current HbA 1c , BMI, activity level, or alcohol usage. Vrabie et al (2018) and 2 studies by Ngo et al (2018) applied a model-based RL algorithm for controlling blood glucose for type 1 Diabetes [22-24]. We used a data-driven approach and considered the blood glucose level feedback from the patient body for training the Q-learning algorithm.…”

Section: Discussionmentioning

confidence: 99%

“…In this method, at each time step t , the algorithm selects a random action with a fixed probability, ε, based on the following formulation. Figure 5 shows the random action selection function, where 0≤ u t ≤1 is a uniform random number drawn at each time step t [23,24].…”

Section: Methodsmentioning

confidence: 99%

“…Ngo et al (2018) used an RL-based algorithm for optimal control of blood glucose in patients with type 1 diabetes using simulations on a combination of the minimum model and part of the Hovorka model [23]. Ngo et al (2018) proposed an RL algorithm for automatically calculating the basal and bolus insulin doses for type 1 diabetes patients using simulation on a blood glucose model with Kalman filter [24].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Reinforcement Learning–Based Method for Management of Type 1 Diabetes: Exploratory Study

et al. 2019

View full text Add to dashboard Cite

Background Type 1 diabetes mellitus (T1DM) is characterized by chronic insulin deficiency and consequent hyperglycemia. Patients with T1DM require long-term exogenous insulin therapy to regulate blood glucose levels and prevent the long-term complications of the disease. Currently, there are no effective algorithms that consider the unique characteristics of T1DM patients to automatically recommend personalized insulin dosage levels. Objective The objective of this study was to develop and validate a general reinforcement learning (RL) framework for the personalized treatment of T1DM using clinical data. Methods This research presents a model-free data-driven RL algorithm, namely Q-learning, that recommends insulin doses to regulate the blood glucose level of a T1DM patient, considering his or her state defined by glycated hemoglobin (HbA1c) levels, body mass index, engagement in physical activity, and alcohol usage. In this approach, the RL agent identifies the different states of the patient by exploring the patient’s responses when he or she is subjected to varying insulin doses. On the basis of the result of a treatment action at time step t, the RL agent receives a numeric reward, positive or negative. The reward is calculated as a function of the difference between the actual blood glucose level achieved in response to the insulin dose and the targeted HbA1c level. The RL agent was trained on 10 years of clinical data of patients treated at the Mass General Hospital. Results A total of 87 patients were included in the training set. The mean age of these patients was 53 years, 59% (51/87) were male, 86% (75/87) were white, and 47% (41/87) were married. The performance of the RL agent was evaluated on 60 test cases. RL agent–recommended insulin dosage interval includes the actual dose prescribed by the physician in 53 out of 60 cases (53/60, 88%). Conclusions This exploratory study demonstrates that an RL algorithm can be used to recommend personalized insulin doses to achieve adequate glycemic control in patients with T1DM. However, further investigation in a larger sample of patients is needed to confirm these findings.

show abstract

“…The resultant desirable states are obtained with the communication between a decision making agent and its environment during PID learning process 32 . Some of the significant elements included in the type‐1 diabetes detection of the proposed framework includes: as indicated in Equation (1), the patient states are included at the time instance k of state vector.

X_{k} = {[g (k) - g_{d} (k) X (k)]}^{T}

…”

Section: Proposed Approachmentioning

confidence: 99%

An adaptive technique based blood glucose control in type‐1 diabetes mellitus patients

Belmon¹,

Auxillia

2020

Numer Methods Biomed Eng

View full text Add to dashboard Cite

This study proposes Grasshopper Optimization Algorithm (GOA) based type 1 diabetes mellitus system utilizing the nonlinear Bergman minimal model with proportional integral derivative (PID) controller. GOA is the optimization algorithm, which is utilized for selecting the optimized tuning parameters of the PID controller also solves the nonlinear system parameter identification problem. The novelty of the proposed study is to stabilize the glucose level in blood for type 1 diabetic patients by infusion of insulin in reduced time with optimal quantity. Without any intervention to the normal activities of patients, the supply of insulin injection and glucose monitoring is performed automatically for type 1 diabetic patients using this controller. In between the measured variable and set point, the difference is calculated by the PID controller to evaluate an error values. In realistic patient oriented conditions, the control performance evaluation, control optimization, and advanced patient modelling should be highly concentrated during the research/analysis on blood glucose control. Evaluation is performed to analyze control performances and implementation is done on Simulink/MATLAB environment. The performance analysis of the type 1 diabetes mellitus system with GOA technique is also discussed and to improve the control performance, to optimize the controller parameters. The simulation results have proved the substantial improvement in the performance of proposed algorithm with the better results achieved than the other conventional controllers such as PSO‐PID and EHO‐PID.

show abstract

Control of Blood Glucose for Type-1 Diabetes by Using Reinforcement Learning with Feedforward Algorithm

Cited by 25 publications

References 18 publications

Reinforcement learning application in diabetes blood glucose control: A systematic review

Reinforcement learning application in diabetes blood glucose control: A systematic review

A Reinforcement Learning–Based Method for Management of Type 1 Diabetes: Exploratory Study

An adaptive technique based blood glucose control in type‐1 diabetes mellitus patients

Contact Info

Product

Resources

About