Rethinking Transfer Learning for Medical Image Classification

Abstract

Transfer learning (TL) with deep convolutional neural networks (DCNNs) has proved successful in medical image classification (MIC). However, the current practice is puzzling, as MIC typically relies only on low-and/or mid-level features that are learned in the bottom layers of DCNNs. Following this intuition, we question the current strategies of TL in MIC. In this paper, we perform careful experimental comparisons between shallow and deep networks for classification on two chest x-ray datasets, using different TL strategies. We find that deep models are not always favorable, and finetuning truncated deep models almost always yields the best performance, especially in data-poor regimes.

Papaver somniferum and Papaver rhoeas Classification Based on Visible Capsule Images Using a Modified MobileNetV3-Small Network with Transfer Learning

“…DCNNs often fail to achieve higher prediction performance with small sample datasets, they are prone to problems such as training difficulty and overfitting [ 9 ], and it is sometimes difficult to obtain a large amount of data with labels. Transfer learning is an efficient strategy to solve image classification problems with small samples [ 32 , 33 , 34 , 42 , 43 ].…”

“…Existing feature extraction capabilities can be leveraged to accelerate and optimize model learning efficiency with the parameters of a neural network model trained on a large image dataset transferred to a target model to aid in the training of a new model, enabling the training of models with higher recognition accuracy using smaller training samples [ 28 ]. Transfer learning can effectively improve the accuracy and robustness of the model, and has been widely used in text processing, [ 29 , 30 , 31 ] image classification [ 32 , 33 , 34 ], collaborative filtering [ 35 , 36 , 37 ], and artificial intelligence planning [ 38 , 39 ].…”

Towards Domain-Aware Transfer Learning for Medical Image Analysis: Opportunities and Challenges

“…Authors have highlighted various non-trivial differences between natural and medical imaging, on account of which they organized several experiments on two datasets (CheXpert and Retina) and found that i) shallow and simpler models can perform equivalent to standard ImageNet models, ii) feature reuse occurs only at initial layers, iii) larger models (both RI and TL) changes less in starting layers after fine-tuning, i.e., over-parameterization for MIA tasks, iii) converged smaller models show similar filters for both RI and TL, i.e., no feature reuse, iv) TL does offer feature independent benefits like convergence speed, even the scale of pretrained weights adapted for RI helps to converge faster. However, recently [9] has challenged some findings of Raghu et al [8] and pointed out three profound limitations of the experiments: 1) Poor evaluation metric-As the datasets are highly imbalanced, the used metric AUC is not a good choice; 2) Unrepresentative target datasets-Both the datasets contain thousands of samples, while MIA datasets usually range from hundreds to few thousands; 3) Rigid TL methods-Alternative strategies like truncated models can also be explored. Peng et al [9] measured AUROC and AURPC performance metrics for classification on two datasets (CheXpert and Covid), and found that TL mostly outperform RI for both shallow and deep models, mainly smaller datasets benefit more.…”

“…However, recently [9] has challenged some findings of Raghu et al [8] and pointed out three profound limitations of the experiments: 1) Poor evaluation metric-As the datasets are highly imbalanced, the used metric AUC is not a good choice; 2) Unrepresentative target datasets-Both the datasets contain thousands of samples, while MIA datasets usually range from hundreds to few thousands; 3) Rigid TL methods-Alternative strategies like truncated models can also be explored. Peng et al [9] measured AUROC and AURPC performance metrics for classification on two datasets (CheXpert and Covid), and found that TL mostly outperform RI for both shallow and deep models, mainly smaller datasets benefit more. They also found that the truncated TL models perform better than conventional and hybrid TL methods.…”

“…Despite the broad applicability of ImageNet-trained models, little work exists studying its meticulous effects in medical imaging. Some recent works [6][7][8][9] have studied the impact of ImageNet-trained TL in MIA and concluded that it only benefits the early convergence. Additionally, the initial layers of the TL architectures are more responsible for knowledge reuse [7,8].…”