Blinded, randomized trial of sonographer versus AI cardiac function assessment

Abstract

Artificial intelligence (AI) has been developed for echocardiography1–3, although it has not yet been tested with blinding and randomization. Here we designed a blinded, randomized non-inferiority clinical trial (ClinicalTrials.gov ID: NCT05140642; no outside funding) of AI versus sonographer initial assessment of left ventricular ejection fraction (LVEF) to evaluate the impact of AI in the interpretation workflow. The primary end point was the change in the LVEF between initial AI or sonographer assessment and final cardiologist assessment, evaluated by the proportion of studies with substantial change (more than 5% change). From 3,769 echocardiographic studies screened, 274 studies were excluded owing to poor image quality. The proportion of studies substantially changed was 16.8% in the AI group and 27.2% in the sonographer group (difference of −10.4%, 95% confidence interval: −13.2% to −7.7%, P < 0.001 for non-inferiority, P < 0.001 for superiority). The mean absolute difference between final cardiologist assessment and independent previous cardiologist assessment was 6.29% in the AI group and 7.23% in the sonographer group (difference of −0.96%, 95% confidence interval: −1.34% to −0.54%, P < 0.001 for superiority). The AI-guided workflow saved time for both sonographers and cardiologists, and cardiologists were not able to distinguish between the initial assessments by AI versus the sonographer (blinding index of 0.088). For patients undergoing echocardiographic quantification of cardiac function, initial assessment of LVEF by AI was non-inferior to assessment by sonographers.

Cardiac MRI: An Alternative Method to Determine the Left Ventricular Function

“…In the future, more and more artificial intelligence (AI) will be used in every aspect of our daily lives, including medical applications. He et al compared the left ventricular ejection fraction echocardiographically using a sonographer vs. artificial intelligence [ 24 ]. They were able to show that AI is not inferior to the measurement of the left ventricular ejection fraction with echocardiography compared to a sonographer.…”

Cardiovascular Disease Prevention Recommendations From an Online Chat-Based AI Model

“…US investment in health informatics research on AI and digital transformation of health care are as important as supporting basic biomedical research and clinical trials. Because US health care-related errors are a leading cause of death 4,5 and are responsible for high levels of avoidable morbidity and care utilization, a major US national commitment to invest in advancing biomedical AI research, with public-private resourcing, is warranted. A multibillion-dollar fund should be developed, financed by the US health information technology industry and federal government, and deployed along with the institutional biomedical science and population/public health research capabilities and assets of the National Institutes of Health and Centers for Disease Control and Prevention.…”

“…Evidence generation around clinical AI integration and implementation needs to be prioritized, particularly clinical trials, as demonstrated by a recent randomized trial of the use of AI in echocardiography. 4 Rigorous multidisciplinary efforts are warranted to understand AI model limitations, structural biases, and oversight needs. Furthermore, privacy concerns around multimodal patient health data, their exposure to tech and AI models, and the relevant medicolegal implications must be considered.…”

Cardiovascular Disease Prevention Recommendations From an Online Chat-Based AI Model—Reply

“…US investment in health informatics research on AI and digital transformation of health care are as important as supporting basic biomedical research and clinical trials. Because US health care-related errors are a leading cause of death 4,5 and are responsible for high levels of avoidable morbidity and care utilization, a major US national commitment to invest in advancing biomedical AI research, with public-private resourcing, is warranted. A multibillion-dollar fund should be developed, financed by the US health information technology industry and federal government, and deployed along with the institutional biomedical science and population/public health research capabilities and assets of the National Institutes of Health and Centers for Disease Control and Prevention.…”

“…As AI models are developed for more health care use cases, there is a concurrent need to systematically evaluate whether they can meaningfully and safely improve current workflows. Evidence generation around clinical AI integration and implementation needs to be prioritized, particularly clinical trials, as demonstrated by a recent randomized trial of the use of AI in echocardiography . Rigorous multidisciplinary efforts are warranted to understand AI model limitations, structural biases, and oversight needs.…”