Interactive Medical Image Segmentation Using Deep Learning With Image-Specific Fine Tuning

Abstract

Convolutional neural networks (CNNs) have achieved state-of-the-art performance for automatic medical image segmentation. However, they have not demonstrated sufficiently accurate and robust results for clinical use. In addition, they are limited by the lack of image-specific adaptation and the lack of generalizability to previously unseen object classes (a.k.a. zero-shot learning). To address these problems, we propose a novel deep learning-based interactive segmentation framework by incorporating CNNs into a bounding box and scribble-based segmentation pipeline. We propose image-specific fine tuning to make a CNN model adaptive to a specific test image, which can be either unsupervised (without additional user interactions) or supervised (with additional scribbles). We also propose a weighted loss function considering network and interaction-based uncertainty for the fine tuning. We applied this framework to two applications: 2-D segmentation of multiple organs from fetal magnetic resonance (MR) slices, where only two types of these organs were annotated for training and 3-D segmentation of brain tumor core (excluding edema) and whole brain tumor (including edema) from different MR sequences, where only the tumor core in one MR sequence was annotated for training. Experimental results show that: 1) our model is more robust to segment previously unseen objects than state-of-the-art CNNs; 2) image-specific fine tuning with the proposed weighted loss function significantly improves segmentation accuracy; and 3) our method leads to accurate results with fewer user interactions and less user time than traditional interactive segmentation methods.

Machine Learning in Fetal Cardiology: What to Expect

“…More recently, Li et al [33] used a DL approach to automatically segment the fetal body and amniotic fluid from 2-D US data. Other examples of DL for segmentation have targeted the fetal brain and lungs [34,35] and these 2 organs plus the placenta and the maternal kidneys from magnetic resonance imaging [36]. Lastly, an ensemble of decision trees has been used to automatically segment fetal brain structures in 3-D US images [37].…”

Bi-Directional Dermoscopic Feature Learning and Multi-Scale Consistent Decision Fusion for Skin Lesion Segmentation

“…color and texture structure from shallow layer) without combining high-level semantic information, thereby limiting the capability of skin lesion localization as well as the generalization of approaches on other medical image segmentation tasks. Recently, deep learning achieves great success [32], [33] and deep learning based segmentation techniques have been reported in skin lesion prediction following their success in other medial image analysis fields, such as multi-modal brain tumor segmentation [34], [35], gland segmentation [36], pulmonary nodule detection [37], and body organs recognition [38]. Gu et al integrated the dense atrous convolution module and residual multi-kernel pooling with encoder-decoder structure for the segmentation of optic disc, retinal vessel, lung, cell contour and OCT layer [39].…”

The tempest in a cubic millimeter: Image-based refinements necessitate the reconstruction of 3D microvasculature from a large series of damaged alternately-stained histological sections

“…Further papers [23,30,34,71,75,87,104,108,112,113] highlight specific applications of machine learning to medical segmentation or further (medical) image processing. Ref.…”