Effect of data leakage in brain MRI classification using 2D convolutional neural networks

Yağış, Ekin; Atnafu, Selamawet Workalemahu; García, Narciso; Marzi, Chiara; Scheda, Riccardo; Giannelli, Marco; Tessa, Carlo; Citi, Luca; Diciotti, Stefano

doi:10.1038/s41598-021-01681-w

Cited by 60 publications

(46 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This calls into question the reliability of the assessments and hindering an objective comparison between research outcomes. This problem has also been demonstrated in 3D magnetic resonance (MR) imaging studies 13 and in digital pathology 25 , where data leakage between the training and testing sets resulted in over-optimistic classification accuracy (>29% slide level classification accuracy in MR studies and up to 41% higher accuracy in digital pathology). Moreover, greater attention should be paid to the structure of datasets made available to the research community to avoid biasing the evaluation of different methods and undermining the usefulness of open-access datasets.…”

Section: Discussionmentioning

confidence: 92%

“…As for the case of many of the reviewed medical image analysis challenges 8 , one aspect that is sometimes missing or not well described is how the testing dataset is generated from the original pool of data. Moreover, there are examples where the preparation of the testing dataset was described, but its overlap with the training set was not considered [9][10][11][12][13] , undermining the reliability of the reported results. Focusing on deep-learning applications for OCT, depending on the acquisition set-up, volumes are usually acquired with micrometer resolution in the x, y and z directions in a restricted field of view, with tissue structures that are alike and affected by similar noise.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Inflation of test accuracy due to data leakage in deep learning-based classification of OCT images

2022

View full text Add to dashboard Cite

In the application of deep learning on optical coherence tomography (OCT) data, it is common to train classification networks using 2D images originating from volumetric data. Given the micrometer resolution of OCT systems, consecutive images are often very similar in both visible structures and noise. Thus, an inappropriate data split can result in overlap between the training and testing sets, with a large portion of the literature overlooking this aspect. In this study, the effect of improper dataset splitting on model evaluation is demonstrated for three classification tasks using three OCT open-access datasets extensively used, Kermany’s and Srinivasan’s ophthalmology datasets, and AIIMS breast tissue dataset. Results show that the classification performance is inflated by 0.07 up to 0.43 in terms of Matthews Correlation Coefficient (accuracy: 5% to 30%) for models tested on datasets with improper splitting, highlighting the considerable effect of dataset handling on model evaluation. This study intends to raise awareness on the importance of dataset splitting given the increased research interest in implementing deep learning on OCT data.

show abstract

Section: Discussionmentioning

confidence: 92%

Section: Introductionmentioning

confidence: 99%

Inflation of test accuracy due to data leakage in deep learning-based classification of OCT images

2022

View full text Add to dashboard Cite

show abstract

“…We repeated the 5-fold CV ten times to compensate for the sampling bias issue. It is essential to underline that, unlike other works ( 74 , 75 ), we performed a patient-based splitting, and thus avoiding results inflated by the phenomenon of data leakage ( 80 ). We used the average value of the AUROC in the validation set to select the best ML/DL frameworks, and evaluated the generalizability on test sets allocated in the hold-out procedure.…”

Section: Discussionmentioning

confidence: 99%

Machine and Deep Learning Prediction Of Prostate Cancer Aggressiveness Using Multiparametric MRI

et al. 2022

Self Cite

View full text Add to dashboard Cite

Prostate cancer (PCa) is the most frequent male malignancy and the assessment of PCa aggressiveness, for which a biopsy is required, is fundamental for patient management. Currently, multiparametric (mp) MRI is strongly recommended before biopsy. Quantitative assessment of mpMRI might provide the radiologist with an objective and noninvasive tool for supporting the decision-making in clinical practice and decreasing intra- and inter-reader variability. In this view, high dimensional radiomics features and Machine Learning (ML) techniques, along with Deep Learning (DL) methods working on raw images directly, could assist the radiologist in the clinical workflow. The aim of this study was to develop and validate ML/DL frameworks on mpMRI data to characterize PCas according to their aggressiveness. We optimized several ML/DL frameworks on T2w, ADC and T2w+ADC data, using a patient-based nested validation scheme. The dataset was composed of 112 patients (132 peripheral lesions with Prostate Imaging Reporting and Data System (PI-RADS) score ≥ 3) acquired following both PI-RADS 2.0 and 2.1 guidelines. Firstly, ML/DL frameworks trained and validated on PI-RADS 2.0 data were tested on both PI-RADS 2.0 and 2.1 data. Then, we trained, validated and tested ML/DL frameworks on a multi PI-RADS dataset. We reported the performances in terms of Area Under the Receiver Operating curve (AUROC), specificity and sensitivity. The ML/DL frameworks trained on T2w data achieved the overall best performance. Notably, ML and DL frameworks trained and validated on PI-RADS 2.0 data obtained median AUROC values equal to 0.750 and 0.875, respectively, on unseen PI-RADS 2.0 test set. Similarly, ML/DL frameworks trained and validated on multi PI-RADS T2w data showed median AUROC values equal to 0.795 and 0.750, respectively, on unseen multi PI-RADS test set. Conversely, all the ML/DL frameworks trained and validated on PI-RADS 2.0 data, achieved AUROC values no better than the chance level when tested on PI-RADS 2.1 data. Both ML/DL techniques applied on mpMRI seem to be a valid aid in predicting PCa aggressiveness. In particular, ML/DL frameworks fed with T2w images data (objective, fast and non-invasive) show good performances and might support decision-making in patient diagnostic and therapeutic management, reducing intra- and inter-reader variability.

show abstract

“…However, these excellent results are due to data leakage, and studies have shown that the incorrect division of the training set and test set is one of the major causes of data leakage. 45 In literatures, [16][17][18] the authors extracted multiple 2D slice images from each 3D MRI image for classification experiments (among them, Billones et al 16 extracted twenty 2D slice images from each 3D MRI image, Sarraf et al 17 extracted all 2D slice images with nonzero average pixels from each 3D MRI image, and Naz et al 18 selected at least three 2D slice images from each 3D MRI image), and these slices were randomly divided into a training set and test set according to a certain ratio. These incorrect data division methods divide different slices of the same subject into a training set and a test set, resulting in data leakage.…”

Section: Classification Results For Data Augmentationmentioning

confidence: 99%

Diagnosis of Alzheimer's disease using structure highlighting key slice stacking and transfer learning

Ban

Zhang

Lao³

2022

Medical Physics

View full text Add to dashboard Cite

Background In recent years, two‐dimensional convolutional neural network (2D CNN) have been widely used in the diagnosis of Alzheimer's disease (AD) based on structural magnetic resonance imaging (sMRI). However, due to the lack of targeted processing of the key slices of sMRI images, the classification performance of the CNN model needs to be improved. Purpose Therefore, in this paper, we propose a key slice processing technique called the structural highlighting key slice stacking (SHKSS) technique, and we apply it to a 2D transfer learning model for AD classification. Methods Specifically, first, 3D MR images were preprocessed. Second, the 2D axial middle‐layer image was extracted from the MR image as a key slice. Then, the image was normalized by intensity and mapped to the red, green, and blue (RGB) space, and histogram specification was performed on the obtained RGB image to generate the final three‐channel image. The final three‐channel image was input into a pretrained CNN model for AD classification. Finally, classification and generalization experiments were conducted to verify the validity of the proposed method. Results The experimental results on the Alzheimer's Disease Neuroimaging Initiative (ADNI) data set show that our SHKSS method can effectively highlight the structural information in MRI slices. Compared with existing key slice processing techniques, our SHKSS method has an average accuracy improvement of at least 26% on the same test data set, and it has better performance and generalization ability. Conclusions Our SHKSS method not only converts single‐channel images into three‐channel images to match the input requirements of the 2D transfer learning model but also highlights the structural information of MRI slices to improve the accuracy of AD diagnosis.

show abstract

Effect of data leakage in brain MRI classification using 2D convolutional neural networks

Cited by 60 publications

References 65 publications

Inflation of test accuracy due to data leakage in deep learning-based classification of OCT images

Inflation of test accuracy due to data leakage in deep learning-based classification of OCT images

Machine and Deep Learning Prediction Of Prostate Cancer Aggressiveness Using Multiparametric MRI

Diagnosis of Alzheimer's disease using structure highlighting key slice stacking and transfer learning

Contact Info

Product

Resources

About