3D Anisotropic Hybrid Network: Transferring Convolutional Features from 2D Images to 3D Anisotropic Volumes

Abstract

While deep convolutional neural networks (CNN) have been successfully applied for 2D image analysis, it is still challenging to apply them to 3D anisotropic volumes, especially when the within-slice resolution is much higher than the between-slice resolution and when the amount of 3D volumes is relatively small. On one hand, direct learning of CNN with 3D convolution kernels suffers from the lack of data and likely ends up with poor generalization; insufficient GPU memory limits the model size or representational power. On the other hand, applying 2D CNN with generalizable features to 2D slices ignores between-slice information. Coupling 2D network with LSTM to further handle the between-slice information is not optimal due to the difficulty in LSTM learning. To overcome the above challenges, we propose a 3D Anisotropic Hybrid Network (AH-Net) that transfers convolutional features learned from 2D images to 3D anisotropic volumes. Such a transfer inherits the desired strong generalization capability for withinslice information while naturally exploiting between-slice information for more effective modelling. The focal loss is further utilized for more effective end-to-end learning. We experiment with the proposed 3D AH-Net on two different medical image analysis tasks, namely lesion detection from a Digital Breast Tomosynthesis volume, and liver and liver tumor segmentation from a Computed Tomography volume and obtain the state-of-the-art results.

V-NAS: Neural Architecture Search for Volumetric Medical Image Segmentation

“…The first two convolutional layers adopt a kernel size 7×7×1 with stride [2, 2, 1] and 1×1×3 with stride [1, 1, 1]. The overall network architecture is effectively verified by [13] while we add the searching process for color blocks to choose between 2D, 3D, and P3D.…”

“…3, we define 3 Decoder cells, composed of the 2D Decoder D 0 , 3D Decoder D 1 , and P3D Decoder D 2 . The Decoder cell is defined as dense blocks, which shows powerful representation ability in [8,13]. The input of the b-th Decoder cell is denoted as x b while the output as x b+1 , which is the input of the (b + 1)-th Decoder cell.…”

“…However, the demanding computation and high GPU consumption of 3D convolutions limit the depth of neural networks and input volume size, which impedes the massive application of 3D convolutions. Recently, the Pseudo-3D (P3D) [17] was introduced to replace 3D convolution k×k×k with two convolutions, i.e., k×k×1 followed by 1×1×k, which can reduce the number of parameters and show good learning ability in [13,25] on anisotropic medical images. However, all the aforementioned existing works choose the network structure empirically, which often impose explicit constraints, i.e., either 2D, 3D or P3D convolutions only, or 2D and 3D convolutions are separate from each other.…”

Cardiac Segmentation of LGE MRI with Noisy Labels

“…Given all pairs of images X and pseudo labelsŶ , we re-sample them to 1 mm 3 isotropic resolution and train an ensemble E of n fully convolutional neural networks to segment the given foreground classes, with P (X)=E(X) standing for the softmax output probability maps for the different classes in the image. Our network architectures follow the encoder-decoder network proposed in [15], named AH-Net, and [5] based on the popular 3D U-Net architecture [3] with residual connections [16], named SegResNet. For training and implementing these neural networks, we used the NVIDIA Clara Train SDK 1 and NVIDIA Tesla V100 GPU with 16 GB memory.…”

“…For training and implementing these neural networks, we used the NVIDIA Clara Train SDK 1 and NVIDIA Tesla V100 GPU with 16 GB memory. As in [15], we initialize AH-Net from ImageNet pretrained weights using a ResNet-18 encoder branch, utilizing anisotropic (3×3×1) kernels in the encoder path in order to make use of pretrained weights from 2D computer vision tasks. While the initial weights are learned from 2D, all convolutions are still applied in a full 3D fashion throughout the network, allowing it to efficiently learn 3D features from the image.…”