Non-Rigid 2D-3D Registration Using Convolutional Autoencoders

Li, Peixin; Pei, Yuru; Guo, Yukun; Ma, Gengyu; Xu, Tianmin; Zha, Hongbin

doi:10.1109/isbi45749.2020.9098602

Cited by 14 publications

(19 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They determined that their model achieved an execution time of 0.1 s, representing an important enhancement against the conventional registration techniques based on intensity; moreover, it achieved effective registrations 79-99% of the time. Li et al [325] introduced a neural network-based approach for the non-rigid 2D-3D registration of the lateral cephalogram and the volumetric cone-beam CT (CBCT) images.…”

Section: Registrationmentioning

confidence: 99%

Review of deep learning: concepts, CNN architectures, challenges, applications, future directions

et al. 2021

View full text Add to dashboard Cite

In the last few years, the deep learning (DL) computing paradigm has been deemed the Gold Standard in the machine learning (ML) community. Moreover, it has gradually become the most widely used computational approach in the field of ML, thus achieving outstanding results on several complex cognitive tasks, matching or even beating those provided by human performance. One of the benefits of DL is the ability to learn massive amounts of data. The DL field has grown fast in the last few years and it has been extensively used to successfully address a wide range of traditional applications. More importantly, DL has outperformed well-known ML techniques in many domains, e.g., cybersecurity, natural language processing, bioinformatics, robotics and control, and medical information processing, among many others. Despite it has been contributed several works reviewing the State-of-the-Art on DL, all of them only tackled one aspect of the DL, which leads to an overall lack of knowledge about it. Therefore, in this contribution, we propose using a more holistic approach in order to provide a more suitable starting point from which to develop a full understanding of DL. Specifically, this review attempts to provide a more comprehensive survey of the most important aspects of DL and including those enhancements recently added to the field. In particular, this paper outlines the importance of DL, presents the types of DL techniques and networks. It then presents convolutional neural networks (CNNs) which the most utilized DL network type and describes the development of CNNs architectures together with their main features, e.g., starting with the AlexNet network and closing with the High-Resolution network (HR.Net). Finally, we further present the challenges and suggested solutions to help researchers understand the existing research gaps. It is followed by a list of the major DL applications. Computational tools including FPGA, GPU, and CPU are summarized along with a description of their influence on DL. The paper ends with the evolution matrix, benchmark datasets, and summary and conclusion.

show abstract

Section: Registrationmentioning

confidence: 99%

Review of deep learning: concepts, CNN architectures, challenges, applications, future directions

et al. 2021

View full text Add to dashboard Cite

show abstract

“…is most pronounced. 2 Indeed, among the seven studies that trained deep CNNs on synthetic data and evaluated in a way that allowed for comparisons between synthetic and real data performance, we found quite substantial performance drops (Miao et al, 2018;Bier et al, 2019;Gao et al, 2020c;Doerr et al, 2020;Gu et al, 2020;Guan et al, 2020;Li et al, 2020). Worse, three studies used different evaluation metrics in synthetic and real experiments so that comparison was not possible (Miao et al, 2016a;Toth et al, 2018;Esfandiari et al, 2021), and perhaps worst, ten studies that trained on synthetic data never even tested (meaningfully) on real data (Hou et al, 2017(Hou et al, , 2018Pei et al, 2017;Xie et al, 2017;Bier et al, 2018;Foote et al, 2019;Guan et al, 2019;Yang and Chen, 2019;Neumann et al, 2020;Zhang et al, 2020).…”

Section: Preserving Improvements Under Domain Shift From Training To Deploymentmentioning

confidence: 92%

“…In our review we found that unsupervised representation learning techniques are a widely adopted technique to reduce the dimensionality of the parameter space while introducing implicit regularization by confining possible solutions to the principal modes of variation across population- or patient-level observations. We identified 12 studies that propose such techniques or use them as part of the registration pipeline ( Brost et al, 2012 ; Lin and Winey, 2012 ; Chou and Pizer, 2013 , 2014 ; Chou et al, 2013 ; Zhao et al, 2014 ; Baka et al, 2015 ; Pei et al, 2017 ; Chen et al, 2018 ; Zhang et al, 2018 ; Foote et al, 2019 ; Li et al, 2020 ; Zhang et al, 2020 ).…”

Section: Systematic Reviewmentioning

confidence: 99%

“…We found that methods designed for cephalometry were distinct from all other approaches as their primary goal is not generally 2D/3D registration, but 3D reconstruction of the skull given a 2D X-ray. Among the papers included in this review, this problem is often formulated as the deformable 2D/3D registration between a lateral X-ray image of the skull and a 3D atlas using a PCA deformation model, the principal components ω D of which are estimated via a prior set of 3D/3D registrations ( Pei et al, 2017 ; Zhang et al, 2018 ; Li et al, 2020 ). Consequently, these methods rely on population-level models and are thus different from methods used for radiation therapy ( Chou et al, 2013 ; Chou and Pizer, 2013 , 2014 ; Zhao et al, 2014 ; Foote et al, 2019 ) and angiography ( Brost et al, 2012 ; Baka et al, 2015 ), which rely on patient-specific models that are built pre- and intra-operatively, respectively.…”

Section: Systematic Reviewmentioning

confidence: 99%

“…Such methods partially or completely skip the step of precise modeling of image formation or similarity, and instead build up the knowledge in a data-driven manner. We have identified 22 studies that describe methods for direct parameter regression ( Chou and Pizer, 2013 , 2014 ; Chou et al, 2013 ; Zhao et al, 2014 ; Mitrovi et al, 2015 ; Wu et al, 2015 ; Miao et al, 2016a ; Miao et al, 2016b ; Hou et al, 2017 ; Pei et al, 2017 ; Xie et al, 2017 ; Hou et al, 2018 ; Miao et al, 2018 ; Toth et al, 2018 ; Zhang et al, 2018 ; Zheng et al, 2018 ; Guan et al, 2019 , 2020 ; Foote et al, 2019 ; Gao et al, 2020c ; Li et al, 2020 ; Xiangqian et al, 2020 ).…”

Section: Systematic Reviewmentioning

confidence: 99%

See 2 more Smart Citations

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

Unberath

Gao

et al. 2021

Front. Robot. AI

View full text Add to dashboard Cite

Image-based navigation is widely considered the next frontier of minimally invasive surgery. It is believed that image-based navigation will increase the access to reproducible, safe, and high-precision surgery as it may then be performed at acceptable costs and effort. This is because image-based techniques avoid the need of specialized equipment and seamlessly integrate with contemporary workflows. Furthermore, it is expected that image-based navigation techniques will play a major role in enabling mixed reality environments, as well as autonomous and robot-assisted workflows. A critical component of image guidance is 2D/3D registration, a technique to estimate the spatial relationships between 3D structures, e.g., preoperative volumetric imagery or models of surgical instruments, and 2D images thereof, such as intraoperative X-ray fluoroscopy or endoscopy. While image-based 2D/3D registration is a mature technique, its transition from the bench to the bedside has been restrained by well-known challenges, including brittleness with respect to optimization objective, hyperparameter selection, and initialization, difficulties in dealing with inconsistencies or multiple objects, and limited single-view performance. One reason these challenges persist today is that analytical solutions are likely inadequate considering the complexity, variability, and high-dimensionality of generic 2D/3D registration problems. The recent advent of machine learning-based approaches to imaging problems that, rather than specifying the desired functional mapping, approximate it using highly expressive parametric models holds promise for solving some of the notorious challenges in 2D/3D registration. In this manuscript, we review the impact of machine learning on 2D/3D registration to systematically summarize the recent advances made by introduction of this novel technology. Grounded in these insights, we then offer our perspective on the most pressing needs, significant open problems, and possible next steps.

show abstract

Deformable registration of lateral cephalogram and cone‐beam computed tomography image

Zhang

Qin

et al. 2021

Medical Physics

Self Cite

View full text Add to dashboard Cite

Purpose: This study aimed to design and evaluate a novel method for the registration of 2D lateral cephalograms and 3D craniofacial cone‐beam computed tomography (CBCT) images, providing patient‐specific 3D structures from a 2D lateral cephalogram without additional radiation exposure. Methods: We developed a cross‐modal deformable registration model based on a deep convolutional neural network. Our approach took advantage of a low‐dimensional deformation field encoding and an iterative feedback scheme to infer coarse‐to‐fine volumetric deformations. In particular, we constructed a statistical subspace of deformation fields and parameterized the nonlinear mapping function from an image pair, consisting of the target 2D lateral cephalogram and the reference volumetric CBCT, to a latent encoding of the deformation field. Instead of the one‐shot registration by the learned mapping function, a feedback scheme was introduced to progressively update the reference volumetric image and to infer coarse‐to‐fine deformations fields, accounting for the shape variations of anatomical structures. A total of 220 clinically obtained CBCTs were used to train and validate the proposed model, among which 120 CBCTs were used to generate a training dataset with 24k paired synthetic lateral cephalograms and CBCTs. The proposed approach was evaluated on the deformable 2D–3D registration of clinically obtained lateral cephalograms and CBCTs from growing and adult orthodontic patients. Results: Strong structural consistencies were observed between the deformed CBCT and the target lateral cephalogram in all criteria. The proposed method achieved state‐of‐the‐art performances with the mean contour deviation of 0.41±0.12 mm on the anterior cranial base, 0.48±0.17 mm on the mandible, and 0.35±0.08 mm on the maxilla, respectively. The mean surface mesh ranged from 0.78 to 0.97 mm on various craniofacial structures, and the LREs ranged from 0.83 to 1.24 mm on the growing datasets regarding 14 landmarks. The proposed iterative feedback scheme handled the structural details and improved the registration. The resultant deformed volumetric image was consistent with the target lateral cephalogram in both 2D projective planes and 3D volumetric space regarding the multicategory craniofacial structures. Conclusions: The results suggest that the deep learning‐based 2D–3D registration model enables the deformable alignment of 2D lateral cephalograms and CBCTs and estimates patient‐specific 3D craniofacial structures.

show abstract

Non-Rigid 2D-3D Registration Using Convolutional Autoencoders

Cited by 14 publications

References 14 publications

Review of deep learning: concepts, CNN architectures, challenges, applications, future directions

Review of deep learning: concepts, CNN architectures, challenges, applications, future directions

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

Deformable registration of lateral cephalogram and cone‐beam computed tomography image

Contact Info

Product

Resources

About