Augmented reality visualization for laparoscopic surgery

Abstract

Abstract. We present the design and a prototype implementation of a three-dimensional visualization system to assist with laparoscopic surgical procedures. The system uses 3D visualization, depth extraction from laparoscopic images, and six degree-of-freedom head and laparoscope tracking to display a merged real and synthetic image in the surgeon's video-see-through head-mounted display. We also introduce a custom design for this display. A digital light projector, a camera, and a conventional laparoscope create a prototype 3D laparoscope that can extract depth and video imagery. Such a system can restore the physician's natural point of view and head motion parallax that are used to understand the 3D structure during open surgery. These cues are not available in conventional laparoscopic surgery due to the displacement of the laparoscopic camera from the physician's viewpoint. The system can also display multiple laparoscopic range imaging data sets to widen the effective field of view of the device. These data sets can be displayed in true 3D and registered to the exterior anatomy of the patient. Much work remains to realize a clinically useful system, notably in the acquisition speed, reconstruction, and registration of the 3D imagery.

Simulation-based design and rapid prototyping of a parallax-free, orthoscopic video see-through head-mounted display

“…A newer design employed separate eye "pods" [2], each containing a monochrome display and a miniature camera. The camera optical path was folded with mirrors, making the device parallax-free.…”

Context specific descriptors for tracking deforming tissue

“…In general, tissue deformation can be estimated using techniques such as intra-operative ultrasound, structured light (Fuchs et al, 1998) (Albitar et al, 2007), eye tracking and time-of-flight cameras (Höller et al, 2008). Most of these approaches, however, require the introduction of additional hardware into the current operating environment.…”

“…Feature density is dictated by the surface of the tissue in terms of homogeneity, texture and salient structural landmarks as illustrated in Figure 1. Structured light (Fuchs et al, 1998) can be used to constrain this problem by projecting patterns onto the surface of the organs, although this alters the surgeon's view unless an invisible frequency band is used. In (Devernay et al, 2001), correlation based dense stereo matching is used to create a 3D model of the heart.…”

Video-endoscopic image analysis for 3D reconstruction of the surgical scene

“…Augmented reality in surgical environments can provide useful information to overcome surgeon's location and orientation difficulties for intra-operative procedures in MIS such as guiding and advising about the proximity of delicate areas with respect to the surgical instruments, something essential [2], [3].…”