Using photo-consistency to register 2D optical images of the human face to a 3D surface model

Clarkson, Matthew J.; Rueckert, Daniel; Hill, Derek L. G.; Hawkes, David J.

doi:10.1109/34.969117

Cited by 54 publications

(40 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The proposed fusion method provides a means of rapidly capturing the intraoperative environment and overlaying it on the preoperative model, facilitating the correlation of the two spaces. In contrast to methods proposed in several previous studies [4][5][6][7], our method only requires the acquisition of a single untracked optical image. In addition, it is based on the image intensity generated using a direct volume rendering technique, which in turn does not require accurate landmark localization or feature extraction and surface reconstruction.…”

Section: Discussionmentioning

confidence: 91%

“…Previous examples from our laboratory of related techniques using an endoscope include those reported by Dey et al [4], who described a method for mapping the endoscopic image onto the 3D shapes extracted from preoperative images via an optically tracked neuro-endoscope, and Szpala et al [5], who presented a method for overlaying a real-time video image on a cardiac model, also via the use of an optically tracked endoscope. Fusion can also be achieved by directly registering an intraoperatively acquired photographic image with the preoperative model, as demonstrated by Clarkson et al [6], who performed an intensity-based 3D-2D registration of optical images from multiple views to a preoperative 3D surface model, and the work of Miga et al [7], in which surfaces captured with a laser range scanner with texture acquired with a digital camera were registered to a preoperative cortical model.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Fusion and visualization of intraoperative cortical images with preoperative models for epilepsy surgical planning and guidance

Wang

Mirsattari

Parrent

et al. 2011

Computer Aided Surgery

View full text Add to dashboard Cite

Objective: During epilepsy surgery it is important for the surgeon to correlate the preoperative cortical morphology (from preoperative images) with the intraoperative environment. Augmented Reality (AR) provides a solution for combining the real environment with virtual models. However, AR usually requires the use of specialized displays, and its effectiveness in the surgery still needs to be evaluated. The objective of this research was to develop an alternative approach to provide enhanced visualization by fusing a direct (photographic) view of the surgical field with the 3D patient model during image guided epilepsy surgery. Materials and Methods: We correlated the preoperative plan with the intraoperative surgical scene, first by a manual landmark-based registration and then by an intensity-based perspective 3D-2D registration for camera pose estimation. The 2D photographic image was then texture-mapped onto the 3D preoperative model using the solved camera pose. In the proposed method, we employ direct volume rendering to obtain a perspective view of the brain image using GPUaccelerated ray-casting. The algorithm was validated by a phantom study and also in the clinical environment with a neuronavigation system. Results: In the phantom experiment, the 3D Mean Registration Error (MRE) was 2.43 AE 0.32 mm with a success rate of 100%. In the clinical experiment, the 3D MRE was 5.15 AE 0.49 mm with 2D in-plane error of 3.30 AE 1.41 mm. A clinical application of our fusion method for enhanced and augmented visualization for integrated image and functional guidance during neurosurgery is also presented. Conclusions: This paper presents an alternative approach to a sophisticated AR environment for assisting in epilepsy surgery, whereby a real intraoperative scene is mapped onto the surface model of the brain. In contrast to the AR approach, this method needs no specialized display equipment. Moreover, it requires minimal changes to existing systems and workflow, and is therefore well suited to the OR environment. In the phantom and in vivo clinical experiments, we demonstrate that the fusion method can achieve a level of accuracy sufficient for the requirements of epilepsy surgery.

show abstract

Section: Discussionmentioning

confidence: 91%

Section: Introductionmentioning

confidence: 99%

Fusion and visualization of intraoperative cortical images with preoperative models for epilepsy surgical planning and guidance

Wang

Mirsattari

Parrent

et al. 2011

Computer Aided Surgery

View full text Add to dashboard Cite

show abstract

“…This is again a very difficult problem and is referred to as head pose estimation [16] and 3D-2D registration [17]. Various approaches to solve this problem either use intensity or facial features (landmarks).…”

Section: D-2d Alignmentmentioning

confidence: 99%

Analysis of 3d Face Reconstruction

Ramasubramanian¹,

Rangaswamy²

2014

Int. Jour. Intel. Elec. Sys.

View full text Add to dashboard Cite

The 3D shape reconstruction, from 2D images is an converse problem, and is therefore mathematically poorly posed. The solution to 3D shape reconstruction problem is to using a model based approach. Our paper presents an analysis by synthesis method for solving3D face reconstruction problems, using the anatomical landmarks and intensity from 2D frontal face images. To improve the quality of 3D shape reconstruction process we incorporate a number of steps in analysis by synthesis framework. First of all, we approached the 3D model construction problem by using rigid and non rigid surface registration. Secondly, we simplified the shape estimation by using multidimensional amoeba optimization to optimize shape parameters while mapping texture directly using 3D-2D alignment. Thirdly, we evaluate the quality of the 3D shape reconstruction in the context of 3D shape error as well as by visual analysis.

show abstract

“…A variational approach is proposed in [12] for the matching of 3D CT data to 2D ultrasound slices that, unlike other volume-to-slice approaches, only uses the given data and relies on a higher order regularization. A method for the matching of photos of human faces with 3D surface models extracted from MRI data using rigid deformations has been suggested in [8]. Recently, a registration method for sparse but highly accurate 3-D line measurements with a surface extracted from volumetric planning data based on the consistent registration idea and higher order regularization was introduced in [2,4].…”

Section: Introductionmentioning

confidence: 99%

Co-registration of intra-operative brain surface photographs and pre-operative MR images

et al. 2014

View full text Add to dashboard Cite

Purpose: Brain shift, the change in configuration of the brain after opening the dura mater, is a significant problem for neuronavigation. Brain structures at intra-operative deformed positions must be matched with corresponding structures in the pre-operative 3D planning data. A method to co-register the cortical surface from intra-operative microscope images with pre-operative MRI segmented data was developed and tested. Methods: Automated classification of sulci on MRI extracted cortical surfaces was tested by comparison with user guided marking of prominent sulci on an intraoperative photography. A variational registration method with a fidelity energy for 3D deformations of the cortical surface in conjunction with a higher order, linear elastic prior energy was used for the actual registration. The minimization of this energy was performed with a regularized gradient descent scheme using finite elements for spatial discretization. The sulcal classification method was tested on eight different clinical MRI data sets by comparison of the deformed MRI scans with intra-operative photographs of the brain surface. Results: User intervention was required for marking sulci on the photos demonstrating the potential for incorporating an automatic classifier. The actual registration was validated first on an artificial testbed. The complete algorithm for the coregistration of actual clinical MRI data was successful for eight different patients. Conclusions: Pre-operative MRI scans can be registered to intra-operative brain surface photographs using a surface-to-surface registration method. This co-registration method has potential applications in neurosurgery, particularly during functional procedures.

show abstract

Using photo-consistency to register 2D optical images of the human face to a 3D surface model

Cited by 54 publications

References 34 publications

Fusion and visualization of intraoperative cortical images with preoperative models for epilepsy surgical planning and guidance

Fusion and visualization of intraoperative cortical images with preoperative models for epilepsy surgical planning and guidance

Analysis of 3d Face Reconstruction

Co-registration of intra-operative brain surface photographs and pre-operative MR images

Contact Info

Product

Resources

About