Association of Artificial Intelligence–Aided Chest Radiograph Interpretation With Reader Performance and Efficiency

Abstract

IMPORTANCEThe efficient and accurate interpretation of radiologic images is paramount. OBJECTIVE To evaluate whether a deep learning-based artificial intelligence (AI) engine used concurrently can improve reader performance and efficiency in interpreting chest radiograph abnormalities. DESIGN, SETTING, AND PARTICIPANTS This multicenter cohort study was conducted from April to November 2021 and involved radiologists, including attending radiologists, thoracic radiology fellows, and residents, who independently participated in 2 observer performance test sessions. The sessions included a reading session with AI and a session without AI, in a randomized crossover manner with a 4-week washout period in between. The AI produced a heat map and the image-level probability of the presence of the referrable lesion. The data used were collected at 2 quaternary academic hospitals in Boston, Massachusetts: Beth Israel Deaconess Medical Center (The Medical Information Mart for Intensive Care Chest X-Ray [MIMIC-CXR]) and Massachusetts General Hospital (MGH). MAIN OUTCOMES AND MEASURESThe ground truths for the labels were created via consensual reading by 2 thoracic radiologists. Each reader documented their findings in a customized report template, in which the 4 target chest radiograph findings and the reader confidence of the presence of each finding was recorded. The time taken for reporting each chest radiograph was also recorded.Sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC) were calculated for each target finding. RESULTSA total of 6 radiologists (2 attending radiologists, 2 thoracic radiology fellows, and 2 residents) participated in the study. The study involved a total of 497 frontal chest radiographs-247 from the MIMIC-CXR data set (demographic data for patients were not available) and 250 chest radiographs from MGH (mean [SD] age, 63 [16] years; 133 men [53.2%])-from adult patients with and without 4 target findings (pneumonia, nodule, pneumothorax, and pleural effusion). The target findings were found in 351 of 497 chest radiographs. The AI was associated with higher sensitivity for all findings compared with the readers (nodule, 0.

AI-Based CXR First Reading: Current Limitations to Ensure Practical Value

“…The problem is that most of the work on assessing the diagnostic accuracy of AI algorithms for CXR indicates metrics obtained by developers on limited datasets in the so-called “laboratory conditions”. As can be seen from recent studies [ 12 , 15 ], the metrics obtained in this way look attractive for the subsequent implementation of such algorithms in clinical practice. Will AI for CXR analysis also work well and demonstrate high diagnostic accuracy metrics in real clinical practice?…”

“…The diagnostic accuracy of the algorithms provided by the developers is quite high [ 7 , 8 , 9 ], reaching the same accuracy for radiologists [ 10 ], and for some solutions even exceeding them [ 11 , 12 ]. As of the beginning of 2023, 29 AI-based software products have European certification for medical use as a medical device (CE MDR/MDD), of which 11 have passed a similar certification in the United States [ 13 ].…”

Opportunities and risks of ChatGPT in medicine, science, and academic publishing: a modern Promethean dilemma

“…AI has a tremendous potential to revolutionize health care and make it more efficient by improving diagnostics, detecting medical errors, and reducing the burden of paperwork ( 3 , 4 ); however, chances are it will never replace physicians. Algorithms perform relatively well on knowledge-based tests despite the lack of domain-specific training; ChatGPT achieved ~ 66% and ~ 72% on Basic Life Support and Advanced Cardiovascular Life Support tests, respectively ( 5 ), and performed at or near the passing threshold on the United States Medical Licensing Exam ( 6 , 7 ).…”

Methods of Visualizing the Results of an Artificial-Intelligence-Based Computer-Aided Detection System for Chest Radiographs: Effect on the Diagnostic Performance of Radiologists

“…For artificial-intelligence-based computer-aided detection (AI-CAD) tools, the primary aim is to enhance the detection performance of interpreting radiologists or physicians [ 2 , 4 , 5 , 6 , 7 ]. Therefore, in addition to intrinsic performance, the method of delivering the results of the analysis to physicians is the key component of an AI-CAD tool to demonstrate its efficacy and value in clinical practice.…”