Multimodal Image Registration with Deep Context Reinforcement Learning

Ma, Kai; Wang, Jiangping; Singh, Vivek Kumar; Tamersoy, Birgi; Chang, Yao‐Jen; Wimmer, Andreas; Chen, Terrence

doi:10.1007/978-3-319-66182-7_28

Cited by 63 publications

(49 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They claimed that the learnt similarity metric outperformed MI and its variant MIND [63]. [92] Spine, Cardiac 3D-3D CT-CBCT Rigid [101] Chest, Abdomen 2D-2D CT-Depth Image Rigid [108] Spine 3D-2D 3DCT-Xray Rigid [183] Spine 3D-2D 3DCT-Xray Rigid [144] Nasopharyngeal 2D-2D MR-CT Rigid with scaling…”

Section: Overview Of Workmentioning

confidence: 99%

Deep learning in medical image registration: a review

et al. 2020

View full text Add to dashboard Cite

This paper presents a review of deep learning (DL) based medical image registration methods. We summarized the latest developments and applications of DL-based registration methods in the medical field. These methods were classified into seven categories according to their methods, functions and popularity. A detailed review of each category was presented, highlighting important contributions and identifying specific challenges. A short assessment was presented following the detailed review of each category to summarize its achievements and future potentials. We provided a comprehensive comparison among DL-based methods for lung and brain registration using benchmark datasets. Lastly, we analyzed the statistics of all the cited works from various aspects, revealing the popularity and future trend of DL-based medical image registration. 1 arXiv:1912.12318v1 [eess.IV] 27 Dec 2019 1. Summarize the latest developments in DL-based medical image registration. 2. Highlight contributions, identify challenges and outline future trends. 3. Provide detailed statistics on recent publications from different perspectives. 2 Deep Learning 2 Convolutional Neural NetworkConvolutional neural network (CNN) is a class of deep neural networks with regularized multilayer perceptron. CNN uses convolution operation in place of general matrix multiplication in simple neural networks. The convolutional filters and operations in CNN make it suitable for visual imagery signal processing. Because of its excellent feature extraction ability, CNN is one of the most successful models for image analysis. Since the breakthrough of AlexNet [79], many variants of CNN have been proposed and have achieved the-state-of-art performances in various image processing tasks. A typical CNN usually consists of multiple convolutional layers, max pooling layers, batch normalization layers, dropout layers, a sigmoid or softmax layer. In each convolutional layer, multiple channels of feature maps were extracted by sliding trainable convolutional kernels across the input feature maps. Hierarchical features with high-level abstraction are extracted using multiple convolutional layers. These feature maps usually go through multiple fully connected layer before reaching the final decision layer. Max pooling layers are often used to reduce the image sizes and to promote spatial invariance of the network. Batch normalization is used to reduce internal covariate shift among the training samples. Weight regularization and dropout layers are used to alleviate data overfitting. The loss function is defined as the difference between the predicted and the target output. CNN is usually trained by minimizing the loss via gradient back propagation using optimization methods. Many different types of network architectures have been proposed to improve the performance of CNN [93]. U-Net proposed by Ronneberger et al. is among one of the most used network architectures [120]. U-Net was originally used to perform neuronal structures segmentation. U-Net adopts symmetrical contractiv...

show abstract

Section: Overview Of Workmentioning

confidence: 99%

Deep learning in medical image registration: a review

et al. 2020

View full text Add to dashboard Cite

show abstract

“…For example, [28] combine momentum-parametrization for LDDMM shooting [29] and neural networks to learn an end-to-end model for registration. Reinforcement learning approaches have also been used to perform image registration [30]. They treat the registration problem as an iterative update of four translation and two rotation parameters so do not handle free form deformations.…”

Section: Related Workmentioning

confidence: 99%

TeTrIS: Template Transformer Networks for Image Segmentation With Shape Priors

Lee

Petersen

Pawlowski

et al. 2019

IEEE Trans. Med. Imaging

108

View full text Add to dashboard Cite

In this paper we introduce and compare different approaches for incorporating shape prior information into neural network based image segmentation. Specifically, we introduce the concept of template transformer networks where a shape template is deformed to match the underlying structure of interest through an end-to-end trained spatial transformer network. This has the advantage of explicitly enforcing shape priors and is free of discretisation artefacts by providing a soft partial volume segmentation. We also introduce a simple yet effective way of incorporating priors in state-of-the-art pixel-wise binary classification methods such as fully convolutional networks and U-net. Here, the template shape is given as an additional input channel, incorporating this information significantly reduces false positives. We report results on synthetic data and sub-voxel segmentation of coronary lumen structures in cardiac computed tomography showing the benefit of incorporating priors in neural network based image segmentation.

show abstract

“…In work [11] and [23], state space is approximated using CNN and thereby neighbouring frames are not considered in the process of decision making. In our work, we utilize CNN to encode image states and a long short-term memory (LSTM) recurrent neural network to encode hidden states between neighbouring frames.…”

Section: State Space Approximation and Action Definitionmentioning

confidence: 99%

Robust Multimodal Image Registration Using Deep Recurrent Reinforcement Learning

Sun

Yao

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The crucial components of a conventional image registration method are the choice of the right feature representations and similarity measures. These two components, although elaborately designed, are somewhat handcrafted using human knowledge. To this end, these two components are tackled in an end-to-end manner via reinforcement learning in this work. Specifically, an artificial agent, which is composed of a combined policy and value network, is trained to adjust the moving image toward the right direction. We train this network using an asynchronous reinforcement learning algorithm, where a customized reward function is also leveraged to encourage robust image registration. This trained network is further incorporated with a lookahead inference to improve the registration capability. The advantage of this algorithm is fully demonstrated by our superior performance on clinical MR and CT image pairs to other state-of-the-art medical image registration methods.

show abstract

Multimodal Image Registration with Deep Context Reinforcement Learning

Cited by 63 publications

References 13 publications

Deep learning in medical image registration: a review

Deep learning in medical image registration: a review

TeTrIS: Template Transformer Networks for Image Segmentation With Shape Priors

Robust Multimodal Image Registration Using Deep Recurrent Reinforcement Learning

Contact Info

Product

Resources

About