Are All Deep Learning Architectures Alike for Point‐of‐Care Ultrasound?: Evidence From a Cardiac Image Classification Model Suggests Otherwise

Abstract

Objectives-Little is known about optimal deep learning (DL) approaches for point-of-care ultrasound (POCUS) applications. We compared 6 popular DL architectures for POCUS cardiac image classification to determine whether an optimal DL architecture exists for future DL algorithm development in POCUS.Methods-We trained 6 convolutional neural networks (CNNs) with a range of complexities and ages (AlexNet, VGG-16, VGG-19, ResNet50, Den-seNet201, and Inception-v4). Each CNN was trained by using images of 5 typical POCUS cardiac views. Images were extracted from 225 publicly available deidentified POCUS cardiac videos. A total of 750,018 individual images were extracted, with 90% used for model training and 10% for cross-validation. The training time and accuracy achieved were tracked. A real-world test of the algorithms was performed on a set of 125 completely new cardiac images. Descriptive statistics, Pearson R values, and κ values were calculated for each CNN.Results-Accuracy ranged from 96% to 85.6% correct for the 6 CNNs. VGG-16, one of the oldest and simplest CNNs, performed best at 96% correct with 232 minutes to train (R = 0.97; κ = 0.95; P < .00001). The worst-performing CNN was the newer DenseNet201, with 85.6% accuracy and 429 minutes to train (R = 0.92; κ = 0.82; P < .00001).Conclusions-Six common image classification DL algorithms showed considerable variability in their accuracy and training time when trained and tested on identical data, suggesting that not all will perform optimally for POCUS DL applications. Contrary to well-established accuracies for CNNs, more modern and deeper algorithms yielded poorer results.

Point-of-Care Ultrasound

“…To overcome this and offer interpretation assistance at the point of care, early investigations involving several convolutional neural networks (CNN) have shown promise. Contrary to what has been found in other studies using artificial intelligence for non-ultrasound images, the simpler rather than more complex CNNs may be more apt to perform well on the greyscale grainy images characteristic of ultrasound [67]. One commercially available product is harnessing POCUS images on a cloud-based storage system to facilitate deep learning that hopefully will eventually be employed to aid the inexperienced user.…”

Artificial intelligence versus expert: a comparison of rapid visual inferior vena cava collapsibility assessment between POCUS experts and a deep learning algorithm

“…Code for VGG‐16 is available from various public sources including github.com. VGG‐16, which won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) competition in 2014, is an early convolutional neural network using 16 layers and has been shown to be superior for ultrasound deep learning applications in prior work 17 . LSTM refers to a structure that incorporates a convolutional neural network into a structure that tracks temporal changes on ultrasound images and is used in non‐medical circles to identify specific action on sports video and even predict outcomes of movements or actions.…”

<p>Harnessing Real-World Data for Regulatory Use and Applying Innovative Applications</p>

“…35 However, before machine learning approaches can be relied upon to produce accurate RWE, the algorithms must be tested, refined, and retested in real-world situations to ensure high accuracy. 37 Challenges with Using AI and RWD Although AI shows promise for application in the healthcare industry, the diverse, complex, and observational nature of RWD presents challenges for data analysis. 9 For example, claims data are typically generated for insurance billing purposes, not adjudicated in terms of data quality, and medical errors can exist.…”

“…Initial attempts to elevate the quality of analysis might include combining multiple data sources, the consideration of doctor shorthand in language processing, and systematic comparisons of machine learning algorithms within or across therapeutic areas. 34,35,37,38 In some countries, data aggregation may pose an equally significant challenge. Legal barriers around data privacy, eg, European Union General Data Protection Regulation (GDPR), 39 practical barriers related to data storage across multiple organizations (ie, data silos), and economic barriers involving lack of incentives for organizations to collaborate and share data, all affect the availability of RWD to which AI tools can be applied.…”