Multicentre external validation of a commercial artificial intelligence software to analyse chest radiographs in health screening environments with low disease prevalence

Kim, Cherry; Yang, Zhou; Hwang, Sung Ho; Oh, Yu-Whan; Kang, Eun Young; Yong, Hwan Seok

doi:10.1007/s00330-022-09315-z

Cited by 17 publications

(7 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A recent study performed an external validation of an AI algorithm capable of classifying chest X-rays normal or abnormal, through a cohort of real images from two primary care centres, and obtained a sensitivity of 0.35 (95% con dence interval (95% CI) 0.25; 0.48) in classifying normal or abnormal X-rays and an AUROC of 0.65 (95% CI 0.63; 0.66). On the other hand, it obtained an AUROC between 0.77-0.99 in several internal validations (31), highlighting that additional training with data from this environment is necessary to improve their performance when using the models in a speci c clinical environment that differs from the training environment. Another study, conducted in primary care centres and in the emergency department of a hospital in the United Kingdom, concluded that the algorithm could accurately and consistently differentiate abnormalities on an image, obtaining a sensitivity between 0.55 and 0.93, and an AUC between 0.88 and 0.99 depending on the condition, detecting 10 of the most prevalent conditions (32).…”

Section: Introductionmentioning

confidence: 99%

External Validation of an Artificial Intelligence Algorithm for the Analysis of Chest X-Rays in the Primary Care Setting

Catalina,

Vidal-Alaball,

Fuster-Casanovas

et al. 2023

Preprint

View full text Add to dashboard Cite

Interpreting chest X-rays is a complex task, and artificial intelligence algorithms for this purpose are currently being developed. It is important to perform external validations of these algorithms in order to implement them. This study therefore aims to externally validate an AI algorithm's diagnoses in real clinical practice, comparing them to a radiologist’s diagnoses. The aim is also to identify diagnoses the algorithm may not have been trained for. A prospective observational study for the external validation of the AI algorithm in a region of Catalonia, comparing the AI algorithm’s diagnosis with that of the reference radiologist, considered the gold standard. The external validation was performed with a sample of 278 images and reports, 51.8% of which showed no radiological abnormalities according to the radiologist's report. Analysing the validity of the AI algorithm, the average accuracy was 0.95 (95% CI 0.92; 0.98), the sensitivity was 0.48 (95% CI 0.30; 0.66) and the specificity was 0.98 (95% CI 0.97; 0.99). The conditions where the algorithm was most sensitive were external, upper abdominal and cardiac and/or valvular implants. On the other hand, the conditions where the algorithm was less sensitive were in the mediastinum, vessels and bone. The AI algorithm has proven to be useful when identifying images with or without conditions. However, in order to be a valuable tool to help and support experts, it requires additional real-world training to enhance its diagnostic capabilities for some of the conditions analysed.

show abstract

Section: Introductionmentioning

confidence: 99%

External Validation of an Artificial Intelligence Algorithm for the Analysis of Chest X-Rays in the Primary Care Setting

Catalina,

Vidal-Alaball,

Fuster-Casanovas

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…6 This finding is further reinforced by recent studies that demonstrated commercially available AI systems that, although backed by excellent results in peer-reviewed publications, significantly underperform in clinical practice. Kim et al 7 show that radiologist diagnosis assisted by AI, still shows unsatisfactory improvements compared to radiologist diagnosis without AI, as well as severely increased reading times due to the false readings of the AI. In the work of Wynants et al, 8 several research flaws were discovered in publicly available Covid-19 prediction models and the concern for investigation of possible bias in AI is raised.…”

Section: Introductionmentioning

confidence: 99%

Robustness evaluation of CAD systems for lung nodule segmentation using clinically relevant image perturbations

Mammadli,

Hellström,

Viviers

et al. 2024

Medical Imaging 2024: Image Processing

View full text Add to dashboard Cite

Lung cancer has both high incidence and mortality rates compared to other cancer types. One important factor for improved patient survival is early detection. Deep learning for lung nodule detection has been extensively studied, as a tool to facilitate clinicians with early nodule detection and classification. Many publications are reporting high detection accuracy and several models have been introduced to clinical practice. However, certain models may have reduced performance in real-world clinical practice. In this study, we introduce a method to assess the robustness of lung nodule detection models. Medically relevant image perturbations are used to assess the robustness of these models. The perturbations include noise and motion perturbations, which have been created in consultation with an expert radiologist to ensure the clinical relevance of the artifacts for thoracic computed tomography (CT) scans. The evaluated models demonstrate robustness to clinically relevant noise simulations, but it shows less resilience to motion artifacts in perturbed CT scans. This robustness evaluation method, incorporating simulated relevant artifacts, can be extended for use in other applications involving the analysis of CT scans.

show abstract

“…A recent study conducting external validation of an AI algorithm designed to classify chest X-rays as normal or abnormal using a cohort of real images from two primary care centres, indicated that additional training with data from these environments was required to enhance the algorithm’s performance, particularly in clinical setting different from its initial training environment 27 .…”

Section: Introductionmentioning

confidence: 99%

Real-world testing of an artificial intelligence algorithm for the analysis of chest X-rays in primary care settings

Miró Catalina,

Vidal-Alaball,

Fuster-Casanovas

et al. 2024

Sci Rep

View full text Add to dashboard Cite

Interpreting chest X-rays is a complex task, and artificial intelligence algorithms for this purpose are currently being developed. It is important to perform external validations of these algorithms in order to implement them. This study therefore aims to externally validate an AI algorithm’s diagnoses in real clinical practice, comparing them to a radiologist’s diagnoses. The aim is also to identify diagnoses the algorithm may not have been trained for. A prospective observational study for the external validation of the AI algorithm in a region of Catalonia, comparing the AI algorithm’s diagnosis with that of the reference radiologist, considered the gold standard. The external validation was performed with a sample of 278 images and reports, 51.8% of which showed no radiological abnormalities according to the radiologist's report. Analysing the validity of the AI algorithm, the average accuracy was 0.95 (95% CI 0.92; 0.98), the sensitivity was 0.48 (95% CI 0.30; 0.66) and the specificity was 0.98 (95% CI 0.97; 0.99). The conditions where the algorithm was most sensitive were external, upper abdominal and cardiac and/or valvular implants. On the other hand, the conditions where the algorithm was less sensitive were in the mediastinum, vessels and bone. The algorithm has been validated in the primary care setting and has proven to be useful when identifying images with or without conditions. However, in order to be a valuable tool to help and support experts, it requires additional real-world training to enhance its diagnostic capabilities for some of the conditions analysed. Our study emphasizes the need for continuous improvement to ensure the algorithm’s effectiveness in primary care.

show abstract

Multicentre external validation of a commercial artificial intelligence software to analyse chest radiographs in health screening environments with low disease prevalence

Cited by 17 publications

References 30 publications

External Validation of an Artificial Intelligence Algorithm for the Analysis of Chest X-Rays in the Primary Care Setting

External Validation of an Artificial Intelligence Algorithm for the Analysis of Chest X-Rays in the Primary Care Setting

Robustness evaluation of CAD systems for lung nodule segmentation using clinically relevant image perturbations

Real-world testing of an artificial intelligence algorithm for the analysis of chest X-rays in primary care settings

Contact Info

Product

Resources

About