Beyond prediction: Off‐target uses of artificial intelligence‐based predictive analytics in a learning health system

Keim‐Malpass, Jessica; Moorman, Liza P.; Monfredi, Oliver; Clark, Matthew; Bourque, Jamieson M.

doi:10.1002/lrh2.10323

Cited by 3 publications

(2 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…CoMET models are logistic regression models with cubic splines based on physiological measurements from continuous cardiorespiratory monitoring data and on EHR elements of vital signs and laboratory tests. The models were trained separately on cardiorespiratory and cardiovascular events of clinical deterioration leading to escalation in care delivery (Moss et al 2017, Ruminski et al 2019, Blackwell et al 2020, Keim-Malpass et al 2022. The predictors of the CoMET score include: (1) cardiorespiratory dynamics measured from continuous electrocardiogram (ECG) (including heart rate (HR) variability, and pairwise cross-correlations between HR and ECG-derived respiratory rate (RR) local dynamics score, coefficient of sample entropy (COSEn), detrended fluctuation analysis (DFA) of heart inter-beat intervals)-all sampled every 2 s; (2) electronic medical record derived parameters (including: vital signs (temperature, HR, blood pressure, RR, SpO 2, ), oxygen flow rate, laboratory results (complete blood count, basic metabolic panel)-all sampled every 15 min.…”

Section: Methodsmentioning

confidence: 99%

Prospective validation of clinical deterioration predictive models prior to intensive care unit transfer among patients admitted to acute care cardiology wards

Keim-Malpass,

Moorman,

Moorman

et al. 2024

Physiol. Meas.

View full text Add to dashboard Cite

Very few predictive models have been externally validated in a prospective cohort following the implementation of an artificial intelligence analytic system. This type of real-world validation is critically important due to the risk of data drift, or changes in data definitions or clinical practices over time, that could impact model performance in contemporaneous real-world cohorts. In this work, we report the model performance of a predictive analytics tool developed before COVID-19 and demonstrate model performance during the COVID-19 pandemic. The analytic system (CoMETⓇ, Nihon Kohden Digital Health Solutions LLC, Irvine, CA) was implemented in a randomized controlled trial that enrolled 10,422 patient visits in a 1:1 display-on display-off design. The CoMET scores were calculated for all patients but only displayed in the display-on arm. Only the control/display-off group is reported here because the scores could not alter care patterns. Of the 5184 visits in the display-off arm, 311 experienced clinical deterioration and care escalation, resulting in transfer to the intensive care unit (ICU), primarily due to respiratory distress. The model performance of CoMET was assessed based on areas under the receiver operating characteristic curve, which ranged from 0.725 to 0.737. The models were well-calibrated, and there were dynamic increases in the model scores in the hours preceding the clinical deterioration events. A hypothetical alerting strategy based on a rise in score and duration of the rise would have had good performance, with a positive predictive value more than 10-fold the event rate. We conclude that predictive statistical models developed five years before study initiation had good model performance despite the passage of time and the impact of the COVID-19 pandemic. 

show abstract

Section: Methodsmentioning

confidence: 99%

Prospective validation of clinical deterioration predictive models prior to intensive care unit transfer among patients admitted to acute care cardiology wards

Keim-Malpass,

Moorman,

Moorman

et al. 2024

Physiol. Meas.

View full text Add to dashboard Cite

show abstract

“…The models were trained separately on cardiorespiratory and cardiovascular events of clinical deterioration leading to escalation in care delivery. (Moss et al 2017, Ruminski et al 2019, Keim-Malpass et al 2022, Blackwell et al 2020)…”

Section: Methodsmentioning

confidence: 99%

Prospective validation of clinical deterioration predictive models prior to intensive care unit transfer among patients admitted to acute care cardiology wards

Keim-Malpass,

Moorman,

Moorman

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Very few predictive models have been externally validated in a prospective cohort following the implementation of an artificial intelligence analytic system. This type of real-world validation is critically important due to the risk of data drift, or changes in data definitions or clinical practices over time, that could impact model performance in contemporaneous real-world cohorts. In this work, we report the model performance of a predictive analytics tool that was developed prior to COVID-19 and demonstrates model performance during the COVID-19 pandemic. The analytic system (CoMET®, Nihon Kohden Digital Health Solutions LLC, Irvine, CA) was implemented in a randomized controlled trial that enrolled 10,422 patient visits in a 1:1 display-on display-off design. The CoMET scores were calculated for all patients but only displayed in the display-on arm. Only the control/display-off group is reported here because the scores could not alter care patterns. Of the 5184 visits in the display-off arm, 311 experienced clinical deterioration and care escalation, resulting in transfer to the intensive care unit (ICU), primarily due to respiratory distress. The model performance of CoMET was assessed based on areas under the receiver operating characteristic curve, which ranged from 0.732 to 0.745. The models were well-calibrated, and there were dynamic increases in the model scores in the hours preceding the clinical deterioration events. A hypothetical alerting strategy based on a rise in score and duration of the rise would have had good performance, with a positive predictive value more than 10-fold the event rate. We conclude that predictive statistical models developed five years before study initiation had good model performance despite the passage of time and the impact of the COVID-19 pandemic. We speculate that some of the model performance’s stability is due to continuous cardiorespiratory monitoring, which should not drift as practices, policies, and patient populations change.Clinical Trial registrationClinicalTrials.govNCT04359641;https://clinicaltrials.gov/ct2/show/NCT04359641.

show abstract

Leveraging artificial intelligence to advance implementation science: potential opportunities and cautions

Trinkley,

An,

Maw

et al. 2024

Implementation Sci

View full text Add to dashboard Cite

Background The field of implementation science was developed to address the significant time delay between establishing an evidence-based practice and its widespread use. Although implementation science has contributed much toward bridging this gap, the evidence-to-practice chasm remains a challenge. There are some key aspects of implementation science in which advances are needed, including speed and assessing causality and mechanisms. The increasing availability of artificial intelligence applications offers opportunities to help address specific issues faced by the field of implementation science and expand its methods. Main text This paper discusses the many ways artificial intelligence can address key challenges in applying implementation science methods while also considering potential pitfalls to the use of artificial intelligence. We answer the questions of “why” the field of implementation science should consider artificial intelligence, for “what” (the purpose and methods), and the “what” (consequences and challenges). We describe specific ways artificial intelligence can address implementation science challenges related to (1) speed, (2) sustainability, (3) equity, (4) generalizability, (5) assessing context and context-outcome relationships, and (6) assessing causality and mechanisms. Examples are provided from global health systems, public health, and precision health that illustrate both potential advantages and hazards of integrating artificial intelligence applications into implementation science methods. We conclude by providing recommendations and resources for implementation researchers and practitioners to leverage artificial intelligence in their work responsibly. Conclusions Artificial intelligence holds promise to advance implementation science methods (“why”) and accelerate its goals of closing the evidence-to-practice gap (“purpose”). However, evaluation of artificial intelligence’s potential unintended consequences must be considered and proactively monitored. Given the technical nature of artificial intelligence applications as well as their potential impact on the field, transdisciplinary collaboration is needed and may suggest the need for a subset of implementation scientists cross-trained in both fields to ensure artificial intelligence is used optimally and ethically.

show abstract

Beyond prediction: Off‐target uses of artificial intelligence‐based predictive analytics in a learning health system

Cited by 3 publications

References 44 publications

Prospective validation of clinical deterioration predictive models prior to intensive care unit transfer among patients admitted to acute care cardiology wards

Prospective validation of clinical deterioration predictive models prior to intensive care unit transfer among patients admitted to acute care cardiology wards

Prospective validation of clinical deterioration predictive models prior to intensive care unit transfer among patients admitted to acute care cardiology wards

Leveraging artificial intelligence to advance implementation science: potential opportunities and cautions

Contact Info

Product

Resources

About