Accuracy assessment methods for physiological model selection toward evaluation of closed-loop controlled medical devices

Abstract

Physiological closed-loop controlled (PCLC) medical devices are complex systems integrating one or more medical devices with a patient’s physiology through closed-loop control algorithms; introducing many failure modes and parameters that impact performance. These control algorithms should be tested through safety and efficacy trials to compare their performance to the standard of care and determine whether there is sufficient evidence of safety for their use in real care setting. With this aim, credible mathematical models have been constructed and used throughout the development and evaluation phases of a PCLC medical device to support the engineering design and improve safety aspects. Uncertainties about the fidelity of these models and ambiguities about the choice of measures for modeling performance need to be addressed before a reliable PCLC evaluation can be achieved. This research develops tools for evaluating the accuracy of physiological models and establishes fundamental measures for predictive capability assessment across different physiological models. As a case study, we built a refined physiological model of blood volume (BV) response by expanding an original model we developed in our prior work. Using experimental data collected from 16 sheep undergoing hemorrhage and fluid resuscitation, first, we compared the calibration performance of the two candidate physiological models, i.e., original and refined, using root-mean-squared error (RMSE), Akiake information criterion (AIC), and a new multi-dimensional approach utilizing normalized features extracted from the fitting error. Compared to the original model, the refined model demonstrated a significant improvement in calibration performance in terms of RMSE (9%, P = 0.03) and multi-dimensional measure (48%, P = 0.02), while a comparable AIC between the two models verified that the enhanced calibration performance in the refined model is not due to data over-fitting. Second, we compared the physiological predictive capability of the two models under three different scenarios: prediction of subject-specific steady-state BV response, subject-specific transient BV response to hemorrhage perturbation, and leave-one-out inter-subject BV response. Results indicated enhanced accuracy and predictive capability for the refined physiological model with significantly larger proportion of measurements that were within the prediction envelope in the transient and leave-one-out prediction scenarios (P < 0.02). All together, this study helps to identify and merge new methods for credibility assessment and physiological model selection, leading to a more efficient process for PCLC medical device evaluation.

Closed-Loop Controlled Fluid Administration Systems: A Comprehensive Scoping Review

“…With regard to future directions in this field, there are several trends that might be expected: Further maturation of the described systems as well as introduction of new ones; Increased adoption of closed-loop controlled fluid administration. The first scenario, where these systems can be safely operated, will probably be the operating room, where constant supervision by an anesthesiologist provides an important safety net; Deeper understanding of fluid dynamics and their translation to ever-more-complex computational models, meant for better accuracy and validity of both controllers and in silico testing platforms [ 18 , 79 , 80 ]; Introduction of new modalities of artificial intelligence, such as reinforcement learning [ 81 , 82 ] and other deep-learning modalities. While there’s increasing use of deep-learning for anesthesia and critical care-related applications [ 83 , 84 ], we have not identified detailed reports on deep-learning-based systems matching our inclusion criteria, meaning we have not identified a system that incorporates deep-learning-based capabilities into a CL system (or, for that matter, a DS system with a feedback loop of repeat evaluations); Continuing formation of a regulatory pipeline dedicated to autonomous and semi-autonomous controlled systems; Increased use of non-invasive sensors in closed-loop fluid administration systems, as their reliability will gradually increase [ 43 , 85 ], as well as artificial intelligence-based advanced sensing modalities (specifically, feature extraction), such as arterial waveform feature analysis [ 77 , 78 ], aimed at providing personalized resuscitation goals; Gradual increase in the degree of automation—from a regulatory standpoint, decision support systems are generally considered safer and easier to approve.…”

“…Deeper understanding of fluid dynamics and their translation to ever-more-complex computational models, meant for better accuracy and validity of both controllers and in silico testing platforms [ 18 , 79 , 80 ];…”

“…Different testing methods include: In silico simulations —These virtual, computer-based tests measure the controller’s ability to respond accurately and robustly in simulated patients’ scenarios. An example for such a testing platform is reported by Bighamian et al [ 18 ]. This method allows for high-throughput tests of the controller at minimal cost and time, using only a computer on existing datasets but cannot test the hardware components; Hardware-in-loop (HIL) testing —A testing method that incorporates hardware components in a physical, manufactured system that simulates a variety of patient scenarios, while measuring the closed-loop controlled system’s performance [ 19 ].…”

“…In silico simulations —These virtual, computer-based tests measure the controller’s ability to respond accurately and robustly in simulated patients’ scenarios. An example for such a testing platform is reported by Bighamian et al [ 18 ]. This method allows for high-throughput tests of the controller at minimal cost and time, using only a computer on existing datasets but cannot test the hardware components;…”

“…Further maturation of the described systems as well as introduction of new ones; • Increased adoption of closed-loop controlled fluid administration. The first scenario, where these systems can be safely operated, will probably be the operating room, where constant supervision by an anesthesiologist provides an important safety net; • Deeper understanding of fluid dynamics and their translation to ever-more-complex computational models, meant for better accuracy and validity of both controllers and in silico testing platforms [18,79,80];…”