Effects of Label Noise on Deep Learning-Based Skin Cancer Classification

Abstract

Recent studies have shown that deep learning is capable of classifying dermatoscopic images at least as well as dermatologists. However, many studies in skin cancer classification utilize non-biopsy-verified training images. This imperfect ground truth introduces a systematic error, but the effects on classifier performance are currently unknown. Here, we systematically examine the effects of label noise by training and evaluating convolutional neural networks (CNN) with 804 images of melanoma and nevi labeled either by dermatologists or by biopsy. The CNNs are evaluated on a test set of 384 images by means of 4-fold cross validation comparing the outputs with either the corresponding dermatological or the biopsy-verified diagnosis. With identical ground truths of training and test labels, high accuracies with 75.03% (95% CI: 74.39-75.66%) for dermatological and 73.80% (95% CI: 73.10-74.51%) for biopsy-verified labels can be achieved. However, if the CNN is trained and tested with different ground truths, accuracy drops significantly to 64.53% (95% CI: 63.12-65.94%, p < 0.01) on a non-biopsy-verified and to 64.24% (95% CI: 62.66-65.83%, p < 0.01) on a biopsy-verified test set. In conclusion, deep learning methods for skin cancer classification are highly sensitive to label noise and future work should use biopsy-verified training images to mitigate this problem.

The Value of Total Body Photography for the Early Detection of Melanoma: A Systematic Review

“…The image analysis was performed by the respective physicians in all reports. Several landmark studies have recently shown that AI performed on par with dermatologists in the distinction of nevi from melanoma based on dermoscopic images [ 43 , 44 , 45 ]. Similarly, future studies are warranted to investigate whether the analysis of 3-D TBP images using AI is also feasible.…”

Editorial: The Emerging Role of Artificial Intelligence in Dermatology

“…Several other barriers exist with regards to the implementation of AI in clinical dermatology, which have been extensively discussed by Gomolin et al as well, including generalizability, standardization, and interpretability (3,(64)(65)(66)(67). To summarize, several AI algorithms are trained using input data from limited populations, thus they may not be effective in patients from different settings or with unique phototypes.…”

Deep Learning Using CT Images to Grade Clear Cell Renal Cell Carcinoma: Development and Validation of a Prediction Model

“…The shifts between two datasets represent that the pattern-recognition abilities acquired from large datasets may not apply well to our medical task. Second is the noisy label problem [ 16 ]. Inevitably, there are always some cancerous lesions that come from high-grade patients but do not exhibit characteristics sufficient to discriminate them from low-grade patients, resulting in the mismatch between the manual labels and the actual labels.…”