Assessment of the effect of a comprehensive chest radiograph deep learning model on radiologist reports and patient outcomes: a real-world observational study

Abstract

ObjectivesArtificial intelligence (AI) algorithms have been developed to detect imaging features on chest X-ray (CXR) with a comprehensive AI model capable of detecting 124 CXR findings being recently developed. The aim of this study was to evaluate the real-world usefulness of the model as a diagnostic assistance device for radiologists.DesignThis prospective real-world multicentre study involved a group of radiologists using the model in their daily reporting workflow to report consecutive CXRs and recording their feedback on level of agreement with the model findings and whether this significantly affected their reporting.SettingThe study took place at radiology clinics and hospitals within a large radiology network in Australia between November and December 2020.ParticipantsEleven consultant diagnostic radiologists of varying levels of experience participated in this study.Primary and secondary outcome measuresProportion of CXR cases where use of the AI model led to significant material changes to the radiologist report, to patient management, or to imaging recommendations. Additionally, level of agreement between radiologists and the model findings, and radiologist attitudes towards the model were assessed.ResultsOf 2972 cases reviewed with the model, 92 cases (3.1%) had significant report changes, 43 cases (1.4%) had changed patient management and 29 cases (1.0%) had further imaging recommendations. In terms of agreement with the model, 2569 cases showed complete agreement (86.5%). 390 (13%) cases had one or more findings rejected by the radiologist. There were 16 findings across 13 cases (0.5%) deemed to be missed by the model. Nine out of 10 radiologists felt their accuracy was improved with the model and were more positive towards AI poststudy.ConclusionsUse of an AI model in a real-world reporting environment significantly improved radiologist reporting and showed good agreement with radiologists, highlighting the potential for AI diagnostic support to improve clinical practice.

Frequency of Missed Findings on Chest Radiographs (CXRs) in an International, Multicenter Study: Application of AI to Reduce Missed Findings

“…Although the AUCs for standalone AI performance reported in our study are lower than those in prior studies [ 24 ], the assessed AI algorithm detected several missed findings not documented in the original radiology reports. The incremental value of AI for interpreting CXRs in our study follows the trends reported in other AI studies [ 23 , 25 ]. The lower AUCs obtained with the assessed AI algorithm for some missed findings in our study are likely related to the fact that missed findings are more likely to be subtle or difficult to detect, and therefore bring an additional level of complexity to AI performance.…”

“…The most frequent and clinically important missed findings included lung nodules and consolidation at all eight participating sites in both India and the US. A high frequency of missed lung nodules on CXRs has also been reported in prior studies [ 23 ]. Apart from the distribution of missed radiographic findings, our study reports on the performance of an AI validation platform (CARPL) and an AI-CXR algorithm (Qure.ai).…”

Charting the potential of brain computed tomography deep learning systems

“…We have observed encouraging signs, however, within a large practice currently using machine learning systems to facilitate radiology interpretation and diagnosis. These signs include subjectively improved concentration levels and a broad acceptance (even amongst early critics) that these systems do offer useful assistance and that their effects are likely to benefit patients [60]. In addition, we have observed renewed enthusiasm amongst radiologists for their clinical work.…”

“…Mature implementation necessitates an evaluation of effective clinical workflow integration and change management. With appropriate risk management and mature development and implementation, applied CTB machine learning systems have the potential to drive substantial clinical benefits and meaningful improvements in healthcare [60].…”

Radiologist’s Role in Artificial Intelligence Era

“…A single AI application for multiple tasks has been scantly reported. 16 Concerning AI performance, radiologists should confirm AI results at the initial AI software implementation in their departments and continue monitoring. 17 Additionally, AI cannot correlate the exam with the patients' clinical results as radiologists can.…”