Monitoring Approaches for a Pediatric Chronic Kidney Disease Machine Learning Model

Morse, Keith; Brown, Conner; Fleming, Scott L.; Todd, Irene; Powell, Austin; Russell, W. Alton; Scheinker, David; Sutherland, Scott M.; Lu, Jonathan; Watkins, Brendan; Shah, Nigam H.; Pageler, Natalie M.; Palma, Jonathan P.

doi:10.1055/s-0042-1746168

Cited by 3 publications

(1 citation statement)

References 38 publications

(54 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most studies documenting temporal model performance have been conducted in registry or research datasets rather than with operational data from models running in real-time clinical settings (7)(8)(9)16). However, the transition from a retrospective research frame to real-time operational implementation may impact performance as input mappings change and the timing data availability shifts (20)(21)(22). To explore performance drift in an operational setting, we evaluated the performance of two models currently implemented in the production EHR system at Vanderbilt University Medical Center (VUMC): a non-proprietary, externally developed model predicting readmission (LACE+) (23) and a locally developed model predicting suicidal behaviors (Vanderbilt Suicide Attempt and Ideation Likelihood model, VSAIL) (24).…”

Section: Performance Drift In Operational Modelsmentioning

confidence: 99%

Open questions and research gaps for monitoring and updating AI-enabled tools in clinical settings

Davis

Walsh²,

Matheny³

2022

Front. Digit. Health

View full text Add to dashboard Cite

As the implementation of artificial intelligence (AI)-enabled tools is realized across diverse clinical environments, there is a growing understanding of the need for ongoing monitoring and updating of prediction models. Dataset shift—temporal changes in clinical practice, patient populations, and information systems—is now well-documented as a source of deteriorating model accuracy and a challenge to the sustainability of AI-enabled tools in clinical care. While best practices are well-established for training and validating new models, there has been limited work developing best practices for prospective validation and model maintenance. In this paper, we highlight the need for updating clinical prediction models and discuss open questions regarding this critical aspect of the AI modeling lifecycle in three focus areas: model maintenance policies, performance monitoring perspectives, and model updating strategies. With the increasing adoption of AI-enabled tools, the need for such best practices must be addressed and incorporated into new and existing implementations. This commentary aims to encourage conversation and motivate additional research across clinical and data science stakeholders.

show abstract

Section: Performance Drift In Operational Modelsmentioning

confidence: 99%

Open questions and research gaps for monitoring and updating AI-enabled tools in clinical settings

Davis

Walsh²,

Matheny³

2022

Front. Digit. Health

View full text Add to dashboard Cite

show abstract

Artificial intelligence in the care of children and adolescents with chronic diseases: a systematic review

Kerth,

Hagemeister,

Bischops

et al. 2024

Eur J Pediatr

View full text Add to dashboard Cite

The integration of artificial intelligence (AI) and machine learning (ML) has shown potential for various applications in the medical field, particularly for diagnosing and managing chronic diseases among children and adolescents. This systematic review aims to comprehensively analyze and synthesize research on the use of AI for monitoring, guiding, and assisting pediatric patients with chronic diseases. Five major electronic databases were searched (Medline, Scopus, PsycINFO, ACM, Web of Science), along with manual searches of gray literature, personal archives, and reference lists of relevant papers. All original studies as well as conference abstracts and proceedings, focusing on AI applications for pediatric chronic disease care were included. Thirty-one studies met the inclusion criteria. We extracted AI method used, study design, population, intervention, and main results. Two researchers independently extracted data and resolved discrepancies through discussion. AI applications are diverse, encompassing, e.g., disease classification, outcome prediction, or decision support. AI generally performed well, though most models were tested on retrospective data. AI-based tools have shown promise in mental health analysis, e.g., by using speech sampling or social media data to predict therapy outcomes for various chronic conditions. Conclusions: While AI holds potential in pediatric chronic disease care, most reviewed studies are small-scale research projects. Prospective clinical implementations are needed to validate its effectiveness in real-world scenarios. Ethical considerations, cultural influences, and stakeholder attitudes should be integrated into future research. What is known:• Artificial Intelligence (AI) will play a more dominant role in medicine and healthcare in the future and many applications are already being developed. What is new:• Our review provides an overview on how AI-driven systems might be able to support children and adolescents with chronic illnesses. • While many applications are being researched, few have been tested on real-world, prospective, clinical data.

show abstract

Self-supervised machine learning using adult inpatient data produces effective models for pediatric clinical prediction tasks

Lemmon,

Guo,

Steinberg

et al. 2023

Journal of the American Medical Informatics Association

View full text Add to dashboard Cite

Objective Development of electronic health records (EHR)-based machine learning models for pediatric inpatients is challenged by limited training data. Self-supervised learning using adult data may be a promising approach to creating robust pediatric prediction models. The primary objective was to determine whether a self-supervised model trained in adult inpatients was noninferior to logistic regression models trained in pediatric inpatients, for pediatric inpatient clinical prediction tasks. Materials and Methods This retrospective cohort study used EHR data and included patients with at least one admission to an inpatient unit. One admission per patient was randomly selected. Adult inpatients were 18 years or older while pediatric inpatients were more than 28 days and less than 18 years. Admissions were temporally split into training (January 1, 2008 to December 31, 2019), validation (January 1, 2020 to December 31, 2020), and test (January 1, 2021 to August 1, 2022) sets. Primary comparison was a self-supervised model trained in adult inpatients versus count-based logistic regression models trained in pediatric inpatients. Primary outcome was mean area-under-the-receiver-operating-characteristic-curve (AUROC) for 11 distinct clinical outcomes. Models were evaluated in pediatric inpatients. Results When evaluated in pediatric inpatients, mean AUROC of self-supervised model trained in adult inpatients (0.902) was noninferior to count-based logistic regression models trained in pediatric inpatients (0.868) (mean difference = 0.034, 95% CI=0.014-0.057; P < .001 for noninferiority and P = .006 for superiority). Conclusions Self-supervised learning in adult inpatients was noninferior to logistic regression models trained in pediatric inpatients. This finding suggests transferability of self-supervised models trained in adult patients to pediatric patients, without requiring costly model retraining.

show abstract

Monitoring Approaches for a Pediatric Chronic Kidney Disease Machine Learning Model

Cited by 3 publications

References 38 publications

Open questions and research gaps for monitoring and updating AI-enabled tools in clinical settings

Open questions and research gaps for monitoring and updating AI-enabled tools in clinical settings

Artificial intelligence in the care of children and adolescents with chronic diseases: a systematic review

Self-supervised machine learning using adult inpatient data produces effective models for pediatric clinical prediction tasks

Contact Info

Product

Resources

About