How Deeply to Fine-Tune a Convolutional Neural Network: A Case Study Using a Histopathology Dataset

Abstract

Accurate classification of medical images is of great importance for correct disease diagnosis. The automation of medical image classification is of great necessity because it can provide a second opinion or even a better classification in case of a shortage of experienced medical staff. Convolutional neural networks (CNN) were introduced to improve the image classification domain by eliminating the need to manually select which features to use to classify images. Training CNN from scratch requires very large annotated datasets that are scarce in the medical field. Transfer learning of CNN weights from another large non-medical dataset can help overcome the problem of medical image scarcity. Transfer learning consists of fine-tuning CNN layers to suit the new dataset. The main questions when using transfer learning are how deeply to fine-tune the network and what difference in generalization that will make. In this paper, all of the experiments were done on two histopathology datasets using three state-of-the-art architectures to systematically study the effect of block-wise fine-tuning of CNN. Results show that fine-tuning the entire network is not always the best option; especially for shallow networks, alternatively fine-tuning the top blocks can save both time and computational power and produce more robust classifiers.

Comparing Stacking Ensemble Techniques to Improve Musculoskeletal Fracture Image Classification

“…As pointed out by [ 19 , 41 ], fine-tuning the CNN architectures will yield better results and converge faster than training a CNN from scratch. In this paper, we fine-tuned all CNN layers using the ImageNet dataset.…”

Classification of Hematoxylin and Eosin‐Stained Breast Cancer Histology Microscopy Images Using Transfer Learning with EfficientNets

“…Choosing the most suitable layer to begin fine-tuning from requires extensive testing. Studies such as [ 12 ] have investigated which block to tune from such that results are optimal. The authors [ 12 ] concluded that fine-tuning the top layers of a network is much more beneficial than the entire network.…”

“…In general, the shortage of available medical experts [ 12 ], the time-consuming quest to reach a final decision on a diagnosis, and the issue of interobserver variability justify the need for a system that can automatically and accurately classify breast cancer histopathology images. Previous approaches to this problem have been relatively successful considering the available data and return adequate classification accuracies but tend to be computationally expensive.…”

Quick Annotator: an open‐source digital pathology based rapid image annotation tool

“…A u‐net consisting of a block depth of five layers and 113,306 parameters was trained on these image patches in an autoencoding fashion to produce a baseline model, a process shown to learn features associated with tissue presentation [ 16 ]. This base model is subsequently fine‐tuned in a supervised fashion to segment the structure of interest, a common approach to help reduce annotated data requirements [ 17 ]. Next, the user viewed all patches processed by this model in a uniform manifold approximation and projection (UMAP [ 18 ]) plot (Figure 1A ).…”