Automatic correction of performance drift under acquisition shift in medical image classification

Abstract

Image-based prediction models for disease detection are sensitive to changes in data acquisition such as the replacement of scanner hardware or updates to the image processing software. The resulting differences in image characteristics may lead to drifts in clinically relevant performance metrics which could cause harm in clinical decision making, even for models that generalise in terms of area under the receiver-operating characteristic curve. We propose Unsupervised Prediction Alignment, a generic automatic recalibration method that requires no ground truth annotations and only limited amounts of unlabelled example images from the shifted data distribution. We illustrate the effectiveness of the proposed method to detect and correct performance drift in mammography-based breast cancer screening and on publicly available histopathology data. We show that the proposed method can preserve the expected performance in terms of sensitivity/specificity under various realistic scenarios of image acquisition shift, thus offering an important safeguard for clinical deployment.

From Theoretical Models to Practical Deployment: A Perspective and Case Study of Opportunities and Challenges in AI-driven Cardiac Auscultation Research for Low-Income Settings

“…This ensures that patients understand subsequent steps and that follow-up support is assured. Regarding C), recent literature has started to explore automated correction for variations in data acquisition, such as when different hardware or software are used. Unsupervised alignment methods are one of the proposed solutions to address this issue [95].…”