Validation Methods to Promote Real-world Applicability of Machine Learning in Medicine

Rafiq, Riyad Bin; Modave, François; Lee, Min Kyung; Albert, Mark V.

doi:10.1145/3441369.3441372

Cited by 12 publications

(13 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Among the strengths of our work is the detailed clinical characterization that allowed the improvement of the genetic-only model. Most importantly, our study design included a validation step in an independent cohort that was completely blind to the model training and to the features selection process; compared to internal cross-validation procedures, the availability of an external independent population allows for an unbiased and not overly-optimistic estimation of the generalizability of the model and reduces the risk of model overfitting, thus increasing the reliability of our findings [ 23 , 37 , 38 ].…”

Section: Discussionmentioning

confidence: 99%

Combining Clinical and Genetic Data to Predict Response to Fingolimod Treatment in Relapsing Remitting Multiple Sclerosis Patients: A Precision Medicine Approach

Ferré

Clarelli

Pignolet

et al. 2023

JPM

View full text Add to dashboard Cite

A personalized approach is strongly advocated for treatment selection in Multiple Sclerosis patients due to the high number of available drugs. Machine learning methods proved to be valuable tools in the context of precision medicine. In the present work, we applied machine learning methods to identify a combined clinical and genetic signature of response to fingolimod that could support the prediction of drug response. Two cohorts of fingolimod-treated patients from Italy and France were enrolled and divided into training, validation, and test set. Random forest training and robust feature selection were performed in the first two sets respectively, and the independent test set was used to evaluate model performance. A genetic-only model and a combined clinical–genetic model were obtained. Overall, 381 patients were classified according to the NEDA-3 criterion at 2 years; we identified a genetic model, including 123 SNPs, that was able to predict fingolimod response with an AUROC= 0.65 in the independent test set. When combining clinical data, the model accuracy increased to an AUROC= 0.71. Integrating clinical and genetic data by means of machine learning methods can help in the prediction of response to fingolimod, even though further studies are required to definitely extend this approach to clinical applications

show abstract

Section: Discussionmentioning

confidence: 99%

Combining Clinical and Genetic Data to Predict Response to Fingolimod Treatment in Relapsing Remitting Multiple Sclerosis Patients: A Precision Medicine Approach

Ferré

Clarelli

Pignolet

et al. 2023

JPM

View full text Add to dashboard Cite

show abstract

“…Despite that, some works mention the need for special care in the statistical evaluation of the training data of the models. Especially when the groups that originate the training data (patients from a specific hospital or people from certain geographical regions, for instance) have distinct characteristics (data-wise), applying that same model to other groups can lead to low model performance (Sun et al, 2022;Rafiq, Modave, Guha, & Albert, 2020).…”

Section: Discussionmentioning

confidence: 99%

“…Another issue pointed out by some works is the need for good model interpretability (Rafiq, Modave, Guha, & Albert, 2020;Harris et al, 2022;Li et al, 2022;Duckworth et al, 2021). ML model interpretability and explainability can help ensure that ML-enabled applications provide coherent and reliable decisions.…”

Section: Discussionmentioning

confidence: 99%

“…It is directly related to QC3, in which works obtained an average of 0.611, the highest score among all quality criteria. It's noticeable when reading the articles that a relevant part of them mentions problems or specificities related to model evaluation in healthcare applications (Shickel et al, 2020;Rafiq, Modave, Guha, & Albert, 2020;Rojas et al, 2022;Fries et al, 2019). One of the most critical issues mentioned is the need to keep data up-to-date to provide input for continuous and consistent updating of ML models.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Machine learning model evaluation on healthcare applications: A Review

Souza,

Barreto,

Macedo

et al. 2023

Interdisciplinarity and Innovation in Scientific Research

View full text Add to dashboard Cite

Machine Learning (ML) models have been applied to solve problems in various fields, which necessarily involves proper evaluation of models to ensure performance. Once deployed, ML models are subject to performance issues, such as those related to changes in data (drift). This type of issue has prompted efforts in model analysis and maintenance, as well as in continual learning, which seeks the ability to continuously learn from a (continuous) stream of data. Therefore, it's important to understand and develop methodologies that can be used to evaluate ML models, making their use in real-world environments feasible. Amongst current areas of application for ML, one that stands out, in particular, is Machine Learning for Healthcare, especially in conjunction with Software for Decision Support of Medical Applications, which presents specific challenges for the evaluation and monitoring of models, particularly given that incorrect prediction or classification can lead to life-threatening situations. This paper presents a systematic literature review that aims at identifying state-of-the-art techniques for evaluating and maintaining ML models for healthcare in effective use in the real world.

show abstract

“…This will cause misjudgment in the classification error. Also, the data sample variability is significantly higher across patients than within patient data variability [ 24 ]. TensorFlow was used to implement the classification in a Python environment.…”

Section: Design and Experimental Setupmentioning

confidence: 99%

A Low-Power Wireless System for Predicting Early Signs of Sudden Cardiac Arrest Incorporating an Optimized CNN Model Implemented on NVIDIA Jetson

Kota

Sharma

Albert

et al. 2023

Sensors

Self Cite

View full text Add to dashboard Cite

The survival rate for sudden cardiac arrest (SCA) is low, and patients with long-term risks of SCA are not adequately alerted. Understanding SCA’s characteristics will be key to developing preventive strategies. Many lives could be saved if SCA’s early onset could be detected or predicted. Monitoring heart signals continuously is essential for diagnosing sporadic cardiac dysfunction. An electrocardiogram (ECG) can be used to continuously monitor heart function without having to go to the hospital. A zeolite-based dry electrode can provide safe on-skin ECG acquisition while the subject is out-of-hospital and facilitate long-term monitoring. To the ECG signal, a low-power 1 μW read-out circuit was designed and implemented in our prior work. However, having long-term ECG monitoring outside the hospital, i.e., high battery life, and low power consumption while transmission and reception of ECG signal are crucial. This paper proposes a prototype with a 10-bit resolution ADC and nRF24L01 transceivers placed 5 m apart. The system uses the 2.4 GHz worldwide ISM frequency band with GFSK modulation to wirelessly transmit digitized ECG bits at 250 kbps data rate to a physician’s computer (or similar) for continuous monitoring of ECG signals; the power consumption is only 11.2 mW and 4.62 mW during transmission and reception, respectively, with a low bit error rate of ≤0.1%. Additionally, a subject-wise cross-validated, three-fold, optimized convolutional neural network (CNN) model using the Physionet-SCA dataset was implemented on NVIDIA Jetson to identify the irregular heartbeats yielding an accuracy of 89% with a run time of 5.31 s. Normal beat classification has an F1 score of 0.94 and a ROC score of 0.886. Thus, this paper integrates the ECG acquisition and processing unit with low-power wireless transmission and CNN model to detect irregular heartbeats.

show abstract

Validation Methods to Promote Real-world Applicability of Machine Learning in Medicine

Cited by 12 publications

References 34 publications

Combining Clinical and Genetic Data to Predict Response to Fingolimod Treatment in Relapsing Remitting Multiple Sclerosis Patients: A Precision Medicine Approach

Combining Clinical and Genetic Data to Predict Response to Fingolimod Treatment in Relapsing Remitting Multiple Sclerosis Patients: A Precision Medicine Approach

Machine learning model evaluation on healthcare applications: A Review

A Low-Power Wireless System for Predicting Early Signs of Sudden Cardiac Arrest Incorporating an Optimized CNN Model Implemented on NVIDIA Jetson

Contact Info

Product

Resources

About