Non-rigid Craniofacial 2D-3D Registration Using CNN-Based Regression

Pei, Yuru; Zhang, Yungeng; Qin, Haifang; Ma, Gengyu; Guo, Yukun; Xu, Tianmin; Zha, Hongbin

doi:10.1007/978-3-319-67558-9_14

Cited by 24 publications

(20 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Kendall et al [7] uses the L2-norm to regress parameters on the Lie algebra se(3) directly, with a β parameter to weight the contribution between rotation and translation. This was similarly performed by authors in [13,17,10,9], who use the predicted parameters for registration tasks. Alternatively, [8,16] reparameterised the pose parameters as projected co-ordinates on a 2D view plane.…”

Section: Introductionmentioning

confidence: 89%

Computing CNN Loss and Gradients for Pose Estimation with Riemannian Geometry

Hou

Miolane²,

Khanal

et al. 2018

Medical Image Computing and Computer Assisted Intervention – MICCAI 2018

View full text Add to dashboard Cite

Pose estimation, i.e. predicting a 3D rigid transformation with respect to a fixed co-ordinate frame in, SE (3), is an omnipresent problem in medical image analysis. Deep learning methods often parameterise poses with a representation that separates rotation and translation. Available frameworks do not provide means to calculate loss on a manifold, regression is usually performed using the L2-norm independently on the rotation's and the translation's parameterisations. This is a metric for linear spaces that does not take into account the Lie group structure of SE(3). We propose a general Riemannian formulation of the pose estimation problem, and train CNNs directly on SE(3) equipped with a left-invariant Riemannian metric. The loss between the ground truth and predicted pose (elements of the manifold) is calculated as the Riemannian geodesic distance, which couples together the translation and rotation components. Network weights are updated by back-propagating the gradient with respect to the predicted pose on the tangent space of the manifold SE(3). We thoroughly evaluate the effectiveness of our loss function by comparing its performance with popular and most commonly used existing methods, on tasks such as image-based localisation and intensity-based 2D/3D registration. We also show that hyper-parameters, used in our loss function to weight the contribution between rotations and translations, can be intrinsically calculated from the dataset to achieve greater performance margins.

show abstract

Section: Introductionmentioning

confidence: 89%

Computing CNN Loss and Gradients for Pose Estimation with Riemannian Geometry

Hou

Miolane²,

Khanal

et al. 2018

Medical Image Computing and Computer Assisted Intervention – MICCAI 2018

View full text Add to dashboard Cite

show abstract

“…In our review we found that unsupervised representation learning techniques are a widely adopted technique to reduce the dimensionality of the parameter space while introducing implicit regularization by confining possible solutions to the principal modes of variation across population- or patient-level observations. We identified 12 studies that propose such techniques or use them as part of the registration pipeline ( Brost et al, 2012 ; Lin and Winey, 2012 ; Chou and Pizer, 2013 , 2014 ; Chou et al, 2013 ; Zhao et al, 2014 ; Baka et al, 2015 ; Pei et al, 2017 ; Chen et al, 2018 ; Zhang et al, 2018 ; Foote et al, 2019 ; Li et al, 2020 ; Zhang et al, 2020 ).…”

Section: Systematic Reviewmentioning

confidence: 99%

“…We found that methods designed for cephalometry were distinct from all other approaches as their primary goal is not generally 2D/3D registration, but 3D reconstruction of the skull given a 2D X-ray. Among the papers included in this review, this problem is often formulated as the deformable 2D/3D registration between a lateral X-ray image of the skull and a 3D atlas using a PCA deformation model, the principal components ω D of which are estimated via a prior set of 3D/3D registrations ( Pei et al, 2017 ; Zhang et al, 2018 ; Li et al, 2020 ). Consequently, these methods rely on population-level models and are thus different from methods used for radiation therapy ( Chou et al, 2013 ; Chou and Pizer, 2013 , 2014 ; Zhao et al, 2014 ; Foote et al, 2019 ) and angiography ( Brost et al, 2012 ; Baka et al, 2015 ), which rely on patient-specific models that are built pre- and intra-operatively, respectively.…”

Section: Systematic Reviewmentioning

confidence: 99%

“…Such methods partially or completely skip the step of precise modeling of image formation or similarity, and instead build up the knowledge in a data-driven manner. We have identified 22 studies that describe methods for direct parameter regression ( Chou and Pizer, 2013 , 2014 ; Chou et al, 2013 ; Zhao et al, 2014 ; Mitrovi et al, 2015 ; Wu et al, 2015 ; Miao et al, 2016a ; Miao et al, 2016b ; Hou et al, 2017 ; Pei et al, 2017 ; Xie et al, 2017 ; Hou et al, 2018 ; Miao et al, 2018 ; Toth et al, 2018 ; Zhang et al, 2018 ; Zheng et al, 2018 ; Guan et al, 2019 , 2020 ; Foote et al, 2019 ; Gao et al, 2020c ; Li et al, 2020 ; Xiangqian et al, 2020 ).…”

Section: Systematic Reviewmentioning

confidence: 99%

“…The importance of regression learning is primarily attributed to the substantially decreased run-time that enables close to real-time tumor tracking in 3D. Methods directed at cephalometry ( Li et al, 2020 ; Zhang et al, 2018 ; Pei et al, 2017 ) are identical in methodology to the radiation therapy methods. As noted in Section 2.5 , most methods here limit themselves to shape estimation and assume that a global rigid alignment is either performed prior or unnecessary.…”

Section: Systematic Reviewmentioning

confidence: 99%

See 2 more Smart Citations

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

Unberath

Gao

et al. 2021

Front. Robot. AI

View full text Add to dashboard Cite

Image-based navigation is widely considered the next frontier of minimally invasive surgery. It is believed that image-based navigation will increase the access to reproducible, safe, and high-precision surgery as it may then be performed at acceptable costs and effort. This is because image-based techniques avoid the need of specialized equipment and seamlessly integrate with contemporary workflows. Furthermore, it is expected that image-based navigation techniques will play a major role in enabling mixed reality environments, as well as autonomous and robot-assisted workflows. A critical component of image guidance is 2D/3D registration, a technique to estimate the spatial relationships between 3D structures, e.g., preoperative volumetric imagery or models of surgical instruments, and 2D images thereof, such as intraoperative X-ray fluoroscopy or endoscopy. While image-based 2D/3D registration is a mature technique, its transition from the bench to the bedside has been restrained by well-known challenges, including brittleness with respect to optimization objective, hyperparameter selection, and initialization, difficulties in dealing with inconsistencies or multiple objects, and limited single-view performance. One reason these challenges persist today is that analytical solutions are likely inadequate considering the complexity, variability, and high-dimensionality of generic 2D/3D registration problems. The recent advent of machine learning-based approaches to imaging problems that, rather than specifying the desired functional mapping, approximate it using highly expressive parametric models holds promise for solving some of the notorious challenges in 2D/3D registration. In this manuscript, we review the impact of machine learning on 2D/3D registration to systematically summarize the recent advances made by introduction of this novel technology. Grounded in these insights, we then offer our perspective on the most pressing needs, significant open problems, and possible next steps.

show abstract

Deformable 3D–2D image registration and analysis of global spinal alignment in long‐length intraoperative spine imaging

et al. 2022

View full text Add to dashboard Cite

Background Spinal deformation during surgical intervention (caused by patient positioning and/or the correction of malalignment) confounds conventional navigation due to the assumptions of rigid transformation. Moreover, the ability to accurately quantify spinal alignment in the operating room would provide an assessment of the surgical product via metrics that correlate with clinical outcomes. Purpose A method for deformable 3D–2D registration of preoperative CT to intraoperative long‐length tomosynthesis images is reported for an accurate 3D evaluation of device placement in the presence of spinal deformation and automated evaluation of global spinal alignment (GSA). Methods Long‐length tomosynthesis (“Long Film,” LF) images were acquired using an O‐arm imaging system (Medtronic, Minneapolis USA). A deformable 3D–2D patient registration was developed using multi‐scale masking (proceeding from the full‐length image to local subvolumes about each vertebra) to transform vertebral labels and planning information from preoperative CT to the LF images. Automatic measurement of GSA (main thoracic kyphosis [MThK] and lumbar lordosis [LL]) was obtained using a spline fit to registered labels. The “Known‐Component Registration” method for device registration was adapted to the multi‐scale process for 3D device localization from orthogonal LF images. The multi‐scale framework was evaluated using a deformable spine phantom in which pedicle screws were inserted, and deformations were induced over a range in LL ∼25°–80°. Further validation was carried out in a cadaver study with implanted pedicle screws and a similar range of spinal deformation. The accuracy of patient and device registration was evaluated in terms of 3D translational error and target registration error, respectively, and the accuracies of automatic GSA measurements were compared to manual annotation. Results Phantom studies demonstrated accurate registration via the multi‐scale framework for all vertebral levels in both the neutral and deformed spine: median (interquartile range, IQR) patient registration error was 1.1 mm (0.7–1.9 mm IQR). Automatic measures of MThK and LL agreed with manual delineation within −1.1° ± 2.2° and 0.7° ± 2.0° (mean and standard deviation), respectively. Device registration error was 0.7 mm (0.4–1.0 mm IQR) at the screw tip and 0.9° (1.0°–1.5°) about the screw trajectory. Deformable 3D–2D registration significantly outperformed conventional rigid registration (p < 0.05), which exhibited device registration errors of 2.1 mm (0.8–4.1 mm) and 4.1° (1.2°–9.5°). Cadaver studies verified performance under realistic conditions, demonstrating patient registration error of 1.6 mm (0.9–2.1 mm); MThK within −4.2° ± 6.8° and LL within 1.7° ± 3.5°; and device registration error of 0.8 mm (0.5–1.9 mm) and 0.7° (0.4°–1.2°) for the multi‐scale deformable method, compared to 2.5 mm (1.0–7.9 mm) and 2.3° (1.6°–8.1°) for rigid registration (p < 0.05). Conclusion The deformable 3D–2D registration framework leverages long‐length intraoperat...

show abstract

Non-rigid Craniofacial 2D-3D Registration Using CNN-Based Regression

Cited by 24 publications

References 13 publications

Computing CNN Loss and Gradients for Pose Estimation with Riemannian Geometry

Computing CNN Loss and Gradients for Pose Estimation with Riemannian Geometry

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

Deformable 3D–2D image registration and analysis of global spinal alignment in long‐length intraoperative spine imaging

Contact Info

Product

Resources

About