Artificial intelligence versus expert: a comparison of rapid visual inferior vena cava collapsibility assessment between POCUS experts and a deep learning algorithm

Abstract

Objectives We sought to create a deep learning algorithm to determine the degree of inferior vena cava (IVC) collapsibility in critically ill patients to enable novice point‐of‐care ultrasound (POCUS) providers. Methods We used publicly available long short term memory (LSTM) deep learning basic architecture that can track temporal changes and relationships in real‐time video, to create an algorithm for ultrasound video analysis. The algorithm was trained on public domain IVC ultrasound videos to improve its ability to recognize changes in varied ultrasound video. A total of 220 IVC videos were used, 10% of the data was randomly used for cross correlation during training. Data were augmented through video rotation and manipulation to multiply effective training data quantity. After training, the algorithm was tested on the 50 new IVC ultrasound video obtained from public domain sources and not part of the data set used in training or cross validation. Fleiss’ κ was calculated to compare level of agreement between the 3 POCUS experts and between deep learning algorithm and POCUS experts. Results There was very substantial agreement between the 3 POCUS experts with κ = 0.65 (95% CI = 0.49–0.81). Agreement between experts and algorithm was moderate with κ = 0.45 (95% CI = 0.33–0.56). Conclusions Our algorithm showed good agreement with POCUS experts in visually estimating degree of IVC collapsibility that has been shown in previously published studies to differentiate fluid responsive from fluid unresponsive septic shock patients. Such an algorithm could be adopted to run in real‐time on any ultrasound machine with a video output, easing the burden on novice POCUS users by limiting their task to obtaining and maintaining a sagittal proximal IVC view and allowing the artificial intelligence make real‐time determinations.

Making Artificial Intelligence Lemonade Out of Data Lemons

“…37 In one example researchers increased low numbers of adult inferior vena cava videos in a study on ML algorithm design for assessment of patient fluid responsiveness when being resuscitated for sepsis in the emergency department or intensive care unit. 16 Video rotations and flips allowed the researchers to convert a 191 video dataset into 3820 videos used for algorithm training. While simply duplicating videos leads to algorithm performance failure due to overfitting, image augmentations such as rotation, inversions, contrast changes, and others can result in a more robust algorithm than would be possible otherwise with limited data.…”

“…We adapted a simple ML architecture previously proven to perform well on medical ultrasound image evaluation, VGG16, as a base architecture running within a long-short-term-memory (LSTM) algorithm for sequential image analysis. 16 A4C videos were obtained, with permission, from the Stanford EchoNet-Dynamic database, a large public echocardiography study database. The database included A4C videos from 10,036 comprehensive echocardiography studies performed at Stanford echocardiography laboratories.…”

The POCUS Consult: How Point of Care Ultrasound Helps Guide Medical Decision Making

“… 151 Blaivas et al developed and applied a deep learning algorithm to augment real-time video interpretation for POCUS-guided assessment of IVC collapsibility and fluid responsiveness. 152 The algorithm was designed to automatically assess whether ultrasound videos demonstrated an IVC collapsibility of ≥25%. After training the algorithm on 220 public domain IVC ultrasound videos, researchers found that it demonstrated good agreement with three POCUS experts κ=0.45 (95% CI = 0.33–0.56).…”

Artificial Intelligence Application on Point-of-Care Ultrasound

“…Chiefly, AI affords the opportunity to auto-calculate inferior vena cava (IVC) collapsibility and left ventricular outflow tract velocity time integral ( LVOT VTI) (7, Figure 1). Although ultrasound-guided IVC collapsibility was once believed to be a strong proxy for fluid status, the propensity for measurement variability between providers has caused its usage to decrease [7]. Similarly, although LVOT VTI is a strong representation of a patient's left ventricular systolic function and provides unique insights into prognosis, the burden of manual calculation and potential for user error has left it historically underutilized [8,9].…”

“…Similarly, although LVOT VTI is a strong representation of a patient's left ventricular systolic function and provides unique insights into prognosis, the burden of manual calculation and potential for user error has left it historically underutilized [8,9]. Top ultrasound manufacturers have begun implementing AI in this setting due to the computational speed and accuracy with which it can auto-calculate these parameters [7]. Ultrasound is also advantageous for accurate approximation of ventricular filling pressures using a series of doppler measurements without the need for invasive Swan-Ganz catherization [10].…”