Deep learning-based 2D/3D registration of an atlas to biplanar X-ray images

Abstract

Purpose The registration of a 3D atlas image to 2D radiographs enables 3D pre-operative planning without the need to acquire costly and high-dose CT-scans. Recently, many deep-learning-based 2D/3D registration methods have been proposed which tackle the problem as a reconstruction by regressing the 3D image immediately from the radiographs, rather than registering an atlas image. Consequently, they are less constrained against unfeasible reconstructions and have no possibility to warp auxiliary data. Finally, they are, by construction, limited to orthogonal projections. Methods We propose a novel end-to-end trainable 2D/3D registration network that regresses a dense deformation field that warps an atlas image such that the forward projection of the warped atlas matches the input 2D radiographs. We effectively take the projection matrix into account in the regression problem by integrating a projective and inverse projective spatial transform layer into the network. Results Comprehensive experiments conducted on simulated DRRs from patient CT images demonstrate the efficacy of the network. Our network yields an average Dice score of 0.94 and an average symmetric surface distance of 0.84 mm on our test dataset. It has experimentally been determined that projection geometries with 80$$^{\circ }$$ ∘ to 100$$^{\circ }$$ ∘ projection angle difference result in the highest accuracy. Conclusion Our network is able to accurately reconstruct patient-specific CT-images from a pair of near-orthogonal calibrated radiographs by regressing a deformation field that warps an atlas image or any other auxiliary data. Our method is not constrained to orthogonal projections, increasing its applicability in medical practices. It remains a future task to extend the network for uncalibrated radiographs.

Synthetic data accelerates the development of generalizable learning-based algorithms for X-ray image analysis

“…In diagnostic medical image analysis, GAN-based synthesis of novel samples has been used to augment available training data for magnetic resonance imaging [43][44][45][46][47][48] , CT 46,49 , ultrasound 50 , retinal [51][52][53] , skin lesion 54,55 and CXR 56 images. In computer-assisted interventions, early successes on the Sim2Real problem include analysis on endoscopic images 3,57-59 and intra-operative X-ray [60][61][62] . The controlled study here validates this approach in the X-ray domain by showing that Sim2Real compares favourably to Real2Real training.…”

In silico simulation: a key enabling technology for next-generation intelligent surgical systems

“…Tool-to-tissue interaction has so far been limited to visual interaction in the projection [111], rather than physical interactions. The target of learning in this context is often the 2D/3D registration of the projective image with a pre-operative CT [35,193] or statistical atlas [200], enabling 3D information to be computed from 2D images. A consistent challenge, due to the ionizing radiation associated with x-ray imaging, is the reduction in the number of acquisitions, according to the as low as reasonably achievable principle [18,19,201].…”

Deep Learning in Medical Image Registration: Introduction and Survey

“…A coarse-fine registration (Himthani et al, 2022;Naik et al, 2022;Saadat et al, 2022;Van Houtte et al,2022) consists of two stages: The first stage is called coarse registration, which aims at finding a fast registration solution but not optimal. That solution is fine-tuned later in the second stage.…”