Multi-vendor evaluation of artificial intelligence as an independent reader for double reading in breast cancer screening on 275,900 mammograms

Abstract

Background Double reading (DR) in screening mammography increases cancer detection and lowers recall rates, but has sustainability challenges due to workforce shortages. Artificial intelligence (AI) as an independent reader (IR) in DR may provide a cost-effective solution with the potential to improve screening performance. Evidence for AI to generalise across different patient populations, screening programmes and equipment vendors, however, is still lacking. Methods This retrospective study simulated DR with AI as an IR, using data representative of real-world deployments (275,900 cases, 177,882 participants) from four mammography equipment vendors, seven screening sites, and two countries. Non-inferiority and superiority were assessed for relevant screening metrics. Results DR with AI, compared with human DR, showed at least non-inferior recall rate, cancer detection rate, sensitivity, specificity and positive predictive value (PPV) for each mammography vendor and site, and superior recall rate, specificity, and PPV for some. The simulation indicates that using AI would have increased arbitration rate (3.3% to 12.3%), but could have reduced human workload by 30.0% to 44.8%. Conclusions AI has potential as an IR in the DR workflow across different screening programmes, mammography equipment and geographies, substantially reducing human reader workload while maintaining or improving standard of care. Trial registration ISRCTN18056078 (20/03/2019; retrospectively registered).

Developing, purchasing, implementing and monitoring AI tools in radiology: Practical considerations. A multi‐society statement from the ACR, CAR, ESR, RANZCR & RSNA

“…AI has undeniably made impressive progress and for many use‐cases can reach diagnostic performance comparable to that of human readers. This has especially been shown in the context of breast cancer screening 65–69 . However, as discussed above, many factors can influence the technical diagnostic performance of AI tools in clinical practice.…”

Artificial intelligence can improve cancer detection in a double reading screening mammography scenario

“…Although BSA has assisted in detecting breast cancers at an early stage and resulting in reduced cancer deaths by 22% 2 , the missed cancer rate of between 15% and 35% remains unacceptably high 3 . Additionally, the increased workload pressures on breast radiologists 4 , especially for double reading, has prompted a review of the economic viability of this model.…”

Comparison of AI-integrated pathways with human-AI interaction for population mammographic screening

“…Several papers have included accurate ground truths for episodes including false negatives (interval cancers) from appropriate follow-up confirmations [6, 10, 14, 17]. Similarly, various aspects of generalisability have been addressed by a handful of studies [6, 9, 10, 18]. Finally, some studies have explored AI-integration with human readers in screening pathways with simulation studies [15, 18–21].…”

“…Similarly, various aspects of generalisability have been addressed by a handful of studies [6, 9, 10, 18]. Finally, some studies have explored AI-integration with human readers in screening pathways with simulation studies [15, 18–21].…”

“…To gain insight into the performance of AI readers in collaboration with human readers and gain clinically-relevant guidance for the implementation of AI readers in screening programs, we conducted detailed simulation studies of AI integrated screening scenarios using our retrospective study cohort. We use the richness and diversity of our cohort combined with careful simulation of reader behaviour (especially that of the third reader in arbitration situations) to substantially improve on previous efforts [15, 18–21] and provide actionable information relevant to current directions for AI implementation [22]. We report on three promising pathway scenarios: (1) the AI reader-replacement scenario where one of the first two readers is replaced by an AI reader in the current two reader with arbitration system; (2) the AI band-pass scenario where the AI reader is used to filter episodes with high-confidence cancer detection scores straight to recall and high-confidence normal scores to no-recall outcomes without human involvement, while the remaining episodes with intermediate scores from the AI reader are passed to the two reader with arbitration system; and (3) the AI triage scenario in which an AI reader sends high detection score episodes (e.g., top decile) for assessment in a two reader with arbitration system and all other episodes to a single reader.…”