Stress testing reveals gaps in clinic readiness of image-based diagnostic artificial intelligence models

Abstract

Artificial intelligence models match or exceed dermatologists in melanoma image classification. Less is known about their robustness against real-world variations, and clinicians may incorrectly assume that a model with an acceptable area under the receiver operating characteristic curve or related performance metric is ready for clinical use. Here, we systematically assessed the performance of dermatologist-level convolutional neural networks (CNNs) on real-world non-curated images by applying computational “stress tests”. Our goal was to create a proxy environment in which to comprehensively test the generalizability of off-the-shelf CNNs developed without training or evaluation protocols specific to individual clinics. We found inconsistent predictions on images captured repeatedly in the same setting or subjected to simple transformations (e.g., rotation). Such transformations resulted in false positive or negative predictions for 6.5–22% of skin lesions across test datasets. Our findings indicate that models meeting conventionally reported metrics need further validation with computational stress tests to assess clinic readiness.

Patient and general public attitudes towards clinical artificial intelligence: a mixed methods systematic review

“…5 Despite the rapid development of AI technology, AI has been implemented in few real-world settings because of the practical challenges of implementation 6 and the absence of validation using metrics other than accuracy; metrics such as calibration and robustness are rarely calculated. 7 To aid AI implementation, randomised trials for interventions involving AI should follow the recently updated Consolidated Standards of Reporting Trials-AI 8 (updated in 2019) and Standard Protocol Items: Recommendations for Interventional Trials-AI 9 (updated in 2020) guide lines to promote trans parency and completeness. Moreover, there is a need to better understand the perspective of patients, who have the most at stake.…”

Re-focusing explainability in medicine

“…For example, one AI algorithm had an excellent performance distinguishing between nevus or melanoma as long as good quality pictures were provided. 28 In this case, providing doctors explanations that the algorithm only functions within these parameters allows them to evaluate its functioning for the general clinical scenario and specific clinical circumstances. These explanations will facilitate doctors’ understanding and justify disregarding AI results, for example, when a picture has lower quality.…”

Toward Generalizability in the Deployment of Artificial Intelligence in Radiology: Role of Computation Stress Testing to Overcome Underspecification

“…By using multiple, well-designed stress tests, modelers can ensure that the produced model is broadly generalizable. Stress testing has already been shown to rule out spurious models in other domains such as dermatology and natural language processing (68,69). D'Amour et al (41) considered three types of stress tests: shifted performance evaluation, contrastive evaluation, and stratified performance evaluation (Fig 2).…”