A Stress Test of Artificial Intelligence: Can Deep Learning Models Trained From Formal Echocardiography Accurately Interpret Point‐of‐Care Ultrasound?

Abstract

Objectives To test if a deep learning (DL) model trained on echocardiography images could accurately segment the left ventricle (LV) and predict ejection fraction on apical 4‐chamber images acquired by point‐of‐care ultrasound (POCUS). Methods We created a dataset of 333 videos from cardiac POCUS exams acquired in the emergency department. For each video we derived two ground‐truth labels. First, we segmented the LV from one image frame and second, we classified the EF as normal, reduced, or severely reduced. We then classified the media's quality as optimal, adequate, or inadequate. With this dataset we tested the accuracy of automated LV segmentation and EF classification by the best‐in‐class echocardiography trained DL model EchoNet‐Dynamic. Results The mean Dice similarity coefficient for LV segmentation was 0.72 (N = 333; 95% CI 0.70–0.74). Cohen's kappa coefficient for agreement between predicted and ground‐truth EF classification was 0.16 (N = 333). The area under the receiver‐operating curve for the diagnosis of heart failure was 0.74 (N = 333). Model performance improved with video quality for the tasks of LV segmentation and diagnosis of heart failure, but was unchanged with EF classification. For all tasks the model was less accurate than the published benchmarks for EchoNet‐Dynamic. Conclusions Performance of a DL model trained on formal echocardiography worsened when challenged with images captured during resuscitations. DL models intended for assessing bedside ultrasound should be trained on datasets composed of POCUS images. Such datasets have yet to be made publicly available.

Validation of machine learning models for estimation of left ventricular ejection fraction on point-of-care ultrasound: insights on features that impact performance

“…The impact of imaging differences in clinical POCUS as compared to echocardiography should not be underestimated. Crockett et al applied a “best-in-class” echocardiography trained ML model, EchoNet-Dynamic, to a retrospective collection of cardiac POCUS studies and found suboptimal ML model performance with an AUC of only 0.74 versus the published benchmark of 0.97 for the classification of LVEF < 50% [ 8 ]. Our model fared slightly better with ICC of 0.77 to 0.84 but was similarly impacted by issues regarding image quality related to scanner and clinical factors.…”

“…Machine learning algorithms (ML) have been shown to estimate the LVEF from echocardiography with a high degree of accuracy [1][2][3][4][5][6][7]. However, there are few studies that validate the performance of ML models for the prediction of LVEF on cardiac POCUS [8,9]. Cardiac POCUS imposes challenges additional to those of cart-based echocardiography, further complicating the quantification of cardiac indices such as LVEF.…”