Novel concept for systematic testing of AI models for MRI acquisition shifts with simulated data

Abstract

Recently published guidelines recommend the validation of models for medical image processing against variations of image acquisition parameters. This work presents a novel concept for systematic tests that investigate a model's performance to MRI acquisition shifts in brain scans of multiple sclerosis patients. Contrast changes of T2-weighted FLAIR images related to changes of two scan parameters (TE, TI) are simulated using artificial data. These images were created from 114 brain MRI, normal tissue segmentation masks and expert MS lesion masks (NAMIC, OpenMS, MSSEG2, ISBI15, LC08). Contrast changes were simulated by using the FLAIR signal equation. The experiments evaluate the F1 scores of models trained on images based on different uniform and non-uniform acquisition parameters when tested on typical acquisition parameter shifts. The acquisition parameter variations of the experiments are based on guidelines for quality assurance and the analysis of protocols and images from different institutions and open datasets. The dependence of the F1 score on both parameters could be approximated by linear functions with R 2 scores of 0.94 to 0.98. Thus, linear functions can be used to model the dependence of the F1 score on the influencing factors, thereby allowing for the derivation of a "save" range of image acquisition parameters to meet a desired performance metric. The segmentation model was more sensitive to changes in TE compared to TI. The maximum lesion F1 loss when applying the models to out-of-distribution data ranged from 0.04 to 0.36 and was significant in all the experiments, even when the model was trained on data representing scans of different contrast (p<0.01). This underlines the need for testing segmentation models against acquisition shifts.

Simulation of acquisition shifts in T2 weighted fluid-attenuated inversion recovery magnetic resonance images to stress test artificial intelligence segmentation networks

“…In previous work, using fully simulated data, we showed that the maximum of the response surface plot and its shape are dependent on the contrast distribution of training and test data. 36 The stress test result can thus be a measure of model analysis and optimization. One has to bear in mind that these extreme points are mathematical constraints, given by the minimum and maximum combinations of TE and TI of real sequences.…”