External Validation of an Ensemble Model for Automated Mammography Interpretation by Artificial Intelligence

Abstract

ImportanceWith a shortfall in fellowship-trained breast radiologists, mammography screening programs are looking toward artificial intelligence (AI) to increase efficiency and diagnostic accuracy. External validation studies provide an initial assessment of how promising AI algorithms perform in different practice settings.ObjectiveTo externally validate an ensemble deep-learning model using data from a high-volume, distributed screening program of an academic health system with a diverse patient population.Design, Setting, and ParticipantsIn this diagnostic study, an ensemble learning method, which reweights outputs of the 11 highest-performing individual AI models from the Digital Mammography Dialogue on Reverse Engineering Assessment and Methods (DREAM) Mammography Challenge, was used to predict the cancer status of an individual using a standard set of screening mammography images. This study was conducted using retrospective patient data collected between 2010 and 2020 from women aged 40 years and older who underwent a routine breast screening examination and participated in the Athena Breast Health Network at the University of California, Los Angeles (UCLA).Main Outcomes and MeasuresPerformance of the challenge ensemble method (CEM) and the CEM combined with radiologist assessment (CEM+R) were compared with diagnosed ductal carcinoma in situ and invasive cancers within a year of the screening examination using performance metrics, such as sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC).ResultsEvaluated on 37 317 examinations from 26 817 women (mean [SD] age, 58.4 [11.5] years), individual model AUROC estimates ranged from 0.77 (95% CI, 0.75-0.79) to 0.83 (95% CI, 0.81-0.85). The CEM model achieved an AUROC of 0.85 (95% CI, 0.84-0.87) in the UCLA cohort, lower than the performance achieved in the Kaiser Permanente Washington (AUROC, 0.90) and Karolinska Institute (AUROC, 0.92) cohorts. The CEM+R model achieved a sensitivity (0.813 [95% CI, 0.781-0.843] vs 0.826 [95% CI, 0.795-0.856]; P = .20) and specificity (0.925 [95% CI, 0.916-0.934] vs 0.930 [95% CI, 0.929-0.932]; P = .18) similar to the radiologist performance. The CEM+R model had significantly lower sensitivity (0.596 [95% CI, 0.466-0.717] vs 0.850 [95% CI, 0.766-0.923]; P < .001) and specificity (0.803 [95% CI, 0.734-0.861] vs 0.945 [95% CI, 0.936-0.954]; P < .001) than the radiologist in women with a prior history of breast cancer and Hispanic women (0.894 [95% CI, 0.873-0.910] vs 0.926 [95% CI, 0.919-0.933]; P = .004).Conclusions and RelevanceThis study found that the high performance of an ensemble deep-learning model for automated screening mammography interpretation did not generalize to a more diverse screening cohort, suggesting that the model experienced underspecification. This study suggests the need for model transparency and fine-tuning of AI models for specific target populations prior to their clinical adoption.

Artificial Intelligence Applications in Breast Imaging: Current Status and Future Directions

“…A study evaluating a well-known, previously externally validated, high performing AI algorithm on an independent, external, diverse population found that certain patient groups had much lower performance compared to other groups and previously published performances. These issues raise concerns about unintended secondary consequences for inadequate inclusion of all patient groups in testing and validation data sets [23].…”

The Need for Targeted Labeling of Machine Learning–Based Software as a Medical Device

“…The FDA has created a framework for adapting current medical device regulation for ML-based software. As the work of Hsu et al 1 show, there is a pressing need for more evaluation of these tools beyond standardized benchmark data sets, and into clinical settings that reflect real patient diversity. Moreover, there is a need to generate targeted labels to allow end users to understand for whom a tool is most appropriate.…”

“…This type of evaluation, however, leaves out the question on how the model will behave for a population with different characteristics and images acquired using different scanners. Hsu et al, 1 following an increasing number of studies of this type, followed a different type of evaluation. They took an ensemble of ML algorithms for mammography initially developed using data from the DREAM Mammography Challenge and evaluated it on an independent large data set with different demographic and clinical characteristics.…”

“…However, there is also a downside where end users-who do not necessarily understand how the underlying ML algorithms operate-need to trust the accuracy of such tools. As the recent paper by Hsu et al 1 reported, sometimes this trust may be unwarranted.…”

“…Therefore, instead of labeling for the tool’s use in broad categories such as adult vs pediatric populations, developers can retrospectively test their software in more targeted populations such as women with dense breast tissue and label their tools appropriately. As seen in the paper by Hsu and colleagues, such targeted assessments are both feasible and vital.…”