Use of artificial intelligence for image analysis in breast cancer screening programmes: systematic review of test accuracy

Abstract

Objective To examine the accuracy of artificial intelligence (AI) for the detection of breast cancer in mammography screening practice. Design Systematic review of test accuracy studies. Data sources Medline, Embase, Web of Science, and Cochrane Database of Systematic Reviews from 1 January 2010 to 17 May 2021. Eligibility criteria Studies reporting test accuracy of AI algorithms, alone or in combination with radiologists, to detect cancer in women’s digital mammograms in screening practice, or in test sets. Reference standard was biopsy with histology or follow-up (for screen negative women). Outcomes included test accuracy and cancer type detected. Study selection and synthesis Two reviewers independently assessed articles for inclusion and assessed the methodological quality of included studies using the QUality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) tool. A single reviewer extracted data, which were checked by a second reviewer. Narrative data synthesis was performed. Results Twelve studies totalling 131 822 screened women were included. No prospective studies measuring test accuracy of AI in screening practice were found. Studies were of poor methodological quality. Three retrospective studies compared AI systems with the clinical decisions of the original radiologist, including 79 910 women, of whom 1878 had screen detected cancer or interval cancer within 12 months of screening. Thirty four (94%) of 36 AI systems evaluated in these studies were less accurate than a single radiologist, and all were less accurate than consensus of two or more radiologists. Five smaller studies (1086 women, 520 cancers) at high risk of bias and low generalisability to the clinical context reported that all five evaluated AI systems (as standalone to replace radiologist or as a reader aid) were more accurate than a single radiologist reading a test set in the laboratory. In three studies, AI used for triage screened out 53%, 45%, and 50% of women at low risk but also 10%, 4%, and 0% of cancers detected by radiologists. Conclusions Current evidence for AI does not yet allow judgement of its accuracy in breast cancer screening programmes, and it is unclear where on the clinical pathway AI might be of most benefit. AI systems are not sufficiently specific to replace radiologist double reading in screening programmes. Promising results in smaller studies are not replicated in larger studies. Prospective studies are required to measure the effect of AI in clinical practice. Such studies will require clear stopping rules to ensure that AI does not reduce programme specificity. Study registration Protocol registered as PROSPERO CRD42020213590.

Correspondence on “Artificial intelligence–assisted phenotype discovery of fragile X syndrome in a population-based sample” by Movaghar et al

“…By advertising a model as AI, it could lead to inflated expectations for patients and health care providers. 2,3 Each patients with FXS were age-and sex-matched with 100 patients without FXS. FXS prevalence in the population is evaluated between 1:7000 and 1:10,000.…”

“…There is a strong need for the patients and their physician to understand how AI identified a disease. 2,3 A trade-off should be made between performance and interpretability, 2,3,7 and performance should be compared with that of highly interpretable models (eg, logistic regression). Transparency and explainability are key to ensure that AI is trusted by patients and their physicians.…”

“…Transparency and explainability are key to ensure that AI is trusted by patients and their physicians. 2,3 In the study by Movaghar et al, 1 the authors provided a short explanation on how random forest works and used a method "to identify variables contributing in the model," but limited details were shared. To ensure the model is trusted and used, full disclosure is required on how the model worked, how outputs were generated, and what the limitations of the model were.…”

“…The model could provide great benefit to patients by "[reducing] the time of clinical diagnosis [of FXS]." However, because many artificial intelligence (AI) systems developed in health care have failed to perform in real life 2,3 (vs. what was initially claimed in scientific publications), a thorough look at the model is needed to ensure it can deliver as promised. In this correspondence, I would like to discuss further the limitations of the model, provide suggestions for improvement, and highlight good quality/validation practices for AI-based recommendation systems in the genetics field.…”

Hitting the Target: Stopping Active Learning at the Cost-Based Optimum

“…Breast cancer is a leading cause of death among women worldwide. Because it is easier to treat the earlier it is detected, a number of countries have introduced screening programmes Freeman et al (2021). Mammograms require one or two radiologists to analyse and are subject to human error with 15% to 35% of cancer cases going undetected Freeman et al (2021).…”