Deep learning of the sectional appearances of 3D CT images for anatomical structure segmentation based on an FCN voting method

Abstract

Purpose: We propose a single network trained by pixel-to-label deep learning to address the general issue of automatic multiple organ segmentation in three-dimensional (3D) computed tomography (CT) images. Our method can be described as a voxel-wise multiple-class classification scheme for automatically assigning labels to each pixel/voxel in a 2D/3D CT image. Methods: We simplify the segmentation algorithms of anatomical structures (including multiple organs) in a CT image (generally in 3D) to a majority voting scheme over the semantic segmentation of multiple 2D slices drawn from different viewpoints with redundancy. The proposed method inherits the spirit of fully convolutional networks (FCNs) that consist of "convolution" and "deconvolution" layers for 2D semantic image segmentation, and expands the core structure with 3D-2D-3D transformations to adapt to 3D CT image segmentation. All parameters in the proposed network are trained pixel-to-label from a small number of CT cases with human annotations as the ground truth. The proposed network naturally fulfills the requirements of multiple organ segmentations in CT cases of different sizes that cover arbitrary scan regions without any adjustment. Results: The proposed network was trained and validated using the simultaneous segmentation of 19 anatomical structures in the human torso, including 17 major organs and two special regions (lumen and content inside of stomach). Some of these structures have never been reported in previous research on CT segmentation. A database consisting of 240 (95% for training and 5% for testing) 3D CT scans, together with their manually annotated ground-truth segmentations, was used in our experiments. The results show that the 19 structures of interest were segmented with acceptable accuracy (88.1% and 87.9% voxels in the training and testing datasets, respectively, were labeled correctly) against the ground truth. Conclusions: We propose a single network based on pixel-to-label deep learning to address the challenging issue of anatomical structure segmentation in 3D CT cases. The novelty of this work is the policy of deep learning of the different 2D sectional appearances of 3D anatomical structures for CT cases and the majority voting of the 3D segmentation results from multiple crossed 2D sections to achieve availability and reliability with better efficiency, generality, and flexibility than conventional segmentation methods, which must be guided by human expertise.

Deep convolutional neural network for segmentation of thoracic organs‐at‐risk using cropped 3D images

“…Possible solutions include resampling the images to lower resolution at the sacrifice of high‐resolution details and edge information; building a shallow network, which will reduce the power of the CNN model; extract smaller patches from the input images for network training, which may also reduce the model accuracy due to the loss of image information. A few studies have been proposed to combine 2D/3D deep learning networks with traditional segmentation algorithms such as 3D majority voting, random walk or atlas‐based methods or use collaborative networks to overcome these issues and improve the segmentation accuracy. However, traditional segmentation algorithms may suffer from robustness issues when combined with deep learning methods as they may worsen the outputs of the neural networks on images with suboptimal image quality or reduced contrast, where the hypothesis for such algorithms is generally no longer valid.…”

A Multi-scale Pyramid of 3D Fully Convolutional Networks for Abdominal Multi-organ Segmentation

“…Deep learning-based multi-organ segmentation in abdominal CT has also been approached recently in works like [6,7]. Most of these methods are based on variants of fully convolutional networks (FCNs) [8] that either employ 2D convolutions on orthogonal cross-sections in a slice-by-slice fashion [3,4,5,9] or 3D convolutions [1,2,7]. A common feature of these segmentation methods is that they are able to extract features useful for image segmentation directly from the training imaging data, which is crucial for the success of deep learning.…”

A novel MRI segmentation method using CNN‐based correction network for MRI‐guided adaptive radiotherapy

“…Medical image segmentation has been studied for decades but remains a challenging problem today . Since the invention of the convolutional neural network (CNN), there have been many attempts to utilize CNNs for various image segmentation tasks . Most of the early methods used a simple “sliding window” approach which has many drawbacks including huge overlap of image patches and repeated convolution for the same pixel .…”

“…Zhou et al. proposed a fully convolutional network with 3D major voting for 3D CT image segmentation of 19 types of targets in the human body . The network was trained using multiple 2D slices and then integrated for 3D classification by major voting.…”