Solo or Ensemble? Choosing a CNN Architecture for Melanoma Classification

Abstract

Convolutional neural networks (CNNs) deliver exceptional results for computer vision, including medical image analysis. With the growing number of available architectures, picking one over another is far from obvious. Existing art suggests that, when performing transfer learning, the performance of CNN architectures on ImageNet correlates strongly with their performance on target tasks. We evaluate that claim for melanoma classification, over 9 CNNs architectures, in 5 sets of splits created on the ISIC Challenge 2017 dataset, and 3 repeated measures, resulting in 135 models. The correlations we found were, to begin with, much smaller than those reported by existing art, and disappeared altogether when we considered only the top-performing networks: uncontrolled nuisances (i.e., splits and randomness) overcome any of the analyzed factors. Whenever possible, the best approach for melanoma classification is still to create ensembles of multiple models. We compared two choices for selecting which models to ensemble: picking them at random (among a pool of highquality ones) vs. using the validation set to determine which ones to pick first. For small ensembles, we found a slight advantage on the second approach but found that random choice was also competitive. Although our aim in this paper was not to maximize performance, we easily reached AUCs comparable to the first place on the ISIC Challenge 2017.

Robust Skin Disease Classification by Distilling Deep Neural Network Ensemble for the Mobile Diagnosis of Herpes Zoster

“…Several methods have been proposed to enhance robustness against corruptions; these include mixed data augmentation [37], adversarial noise training [38], and assembling CNN techniques with KD [39]. However, in the skin lesion analysis domain, wherein random nuisances are observed, the correlation between the network and metrics is difficult to analyze from the ImageNet perspective [26]. Furthermore, methods that improve corruption robustness have not yet been extensively tested for a skin lesion analysis.…”

“…Perez. et al conducted extensive experiments for melanoma classification [26] with 135 models, and they showed that the ensembles of neural networks perform significantly better than a single model, even when the models used for the ensemble are randomly selected. However, the ensemble of the neural networks leads to a heavy and larger model, which is infeasible for mobile on-device inference and mobile diagnostic services.…”

An Ensemble Wasserstein Generative Adversarial Network Method for Road Extraction From High Resolution Remote Sensing Images in Rural Areas

“…Due to the spectral heterogeneity in HR-RS images and the occlusions on roads, mistakes of road extraction are inevitable even for the state-of-the-art models. Creating ensembles of multi-models is proved to be an effective way of improving accuracies in CNN methods, because models trained with different architectures or hyper-parameters may have complementary information [40][41][42].…”

“…In addition, his work only uses the U-Net [12] as G which can be improved by other state-of-the-art segmentation models such as BiSeNet [22]. And to further improve the overall performance in CNN methods, creating ensembles of multi-models may be an effective way [40][41][42].…”

The Possibility of Deep Learning-Based, Computer-Aided Skin Tumor Classifiers

“…AI classifiers for the image classification field have been dramatically improved and made more popular by the introduction of CNN. Many strategies, such as creating ensembles of multiple models (62, 63) or using additional information other than image labels, to improve the accuracy of the classifier outside of increasing the number of images for training have been reported (64). Some studies have reported that CNN algorithms have already surpassed the classification efficacy of dermatologists and, in the near future, AI classifiers may gain sufficient sensitivity and specificity to bear the screening burden for detecting malignant skin tumors.…”