A CNN Regression Approach for Real-Time 2D/3D Registration

Abstract

In this paper, we present a Convolutional Neural Network (CNN) regression approach to address the two major limitations of existing intensity-based 2-D/3-D registration technology: 1) slow computation and 2) small capture range. Different from optimization-based methods, which iteratively optimize the transformation parameters over a scalar-valued metric function representing the quality of the registration, the proposed method exploits the information embedded in the appearances of the digitally reconstructed radiograph and X-ray images, and employs CNN regressors to directly estimate the transformation parameters. An automatic feature extraction step is introduced to calculate 3-D pose-indexed features that are sensitive to the variables to be regressed while robust to other factors. The CNN regressors are then trained for local zones and applied in a hierarchical manner to break down the complex regression task into multiple simpler sub-tasks that can be learned separately. Weight sharing is furthermore employed in the CNN regression model to reduce the memory footprint. The proposed approach has been quantitatively evaluated on 3 potential clinical applications, demonstrating its significant advantage in providing highly accurate real-time 2-D/3-D registration with a significantly enlarged capture range when compared to intensity-based methods.

A survey on deep learning in medical image analysis

“…The resultant feature vectors are used to drive the HAMMER registration algorithm instead of handcrafted features. Miao et al (2016) and Yang et al (2016d) used deep learning algorithms to directly predict the registration transform parameters given input images. Miao et al (2016) leveraged CNNs to perform 3D model to 2D xray registration to assess the pose and location of an implanted object during surgery.…”

“…Prostate segmentation in MRI Liao et al (2013) Application of stacked independent subspace analysis networks Cheng et al (2016b) CNN produces energy map for 2D slice based active appearance segmentation Guo et al (2016) Stacked sparse auto-encoders extract features from patches, input to atlas matching and a deformable model Milletari et al (2016b) 3D U-net based CNN architecture with objective function that directly optimizes Dice coefficient, ranks #5 in PROMISE12 Yu et al (2017) 3D fully convolutional network, hybrid between a ResNet and U-net architecture, ranks #1 on PROMISE12 Chen et al (2015c) CT Vertebrae localization; joint learning of vertebrae appearance and dependency on neighbors using CNN Roth et al (2015c) CT Sclerotic metastases detection; random 2D views are analyzed by CNN and aggregated Shen et al (2015a) CT Vertebrae localization and segmentation; CNN for segmenting vertebrae and for center detection Suzani et al (2015) MRI Vertebrae localization, identification and segmentation of vertebrae; CNN used for initial localization Yang et al (2015) MRI Anatomical landmark detection; uses CNN for slice classification for presence of landmark Antony et al (2016) X-ray Osteoarthritis grading; pre-trained ImageNet CNN fine-tuned on knee X-rays Cai et al (2016b) CT, MRI Vertebrae localization; RBM determines position, orientation and label of vertebrae Golan et al (2016) US Hip dysplasia detection; CNN with adversarial component detects structures and performs measurements Korez et al (2016) MRI Vertebral bodies segmentation; voxel probabilities obtained with a 3D CNN are input to deformable model Jamaludin et al (2016) MRI Automatic spine scoring; VGG-19 CNN analyzes vertebral discs and finds lesion hotspots Miao et al (2016) X-ray Total Knee Arthroplasty kinematics by real-time 2D/3D registration using CNN Roth et al (2016c) CT Posterior-element fractures detection; CNN for 2.5D patch-based analysiš Stern et al (2016) MRI Hand age estimation; 2D regression CNN analyzes 13 bones Forsberg et al (2017) MRI Vertebrae detection and labeling; outputs of two CNNs are input to graphical model Spampinato et al (2017) X-ray Skeletal bone age assessment; comparison among several deep learning approaches for the task at hand A surprising number of complete applications with promising results are available; one that stands out is Jamaludin et al (2016) who trained their system with 12K discs and claimed near-human performances across four different radiological scoring ta...…”

Guest Editorial Deep Learning in Medical Imaging: Overview and Future Promise of an Exciting New Technique

“…In Miao et al [34], a CNN regression approach is presented, for real-time 2-D/3-D registration. Three algorithmic strategies are proposed to simplify the underlying mapping to be regressed, and to design a CNN regression model with strong non-linear modeling.…”

Prospects of deep learning for medical imaging

“…The CNN-based approaches could be used as an alternative for the difficult registration problem by finding landmarks or control points through CNN architecture. Miao et al [64] improved the registration in terms of the computation time and capture range using a real-time 2D/3D registration framework based on CNN. Another study proposed a technique for correcting respiratory motion during free-breathing MRI using CNN [65].…”