Automatic localization of endoscope in intraoperative CT image: A simple approach to augmented reality guidance in laparoscopic surgery

Abstract

The use of augmented reality in minimally invasive surgery has been the subject of much research for more than a decade. The endoscopic view of the surgical scene is typically augmented with a 3D model extracted from a preoperative acquisition. However, the organs of interest often present major changes in shape and location because of the pneumoperitoneum and patient displacement. There have been numerous attempts to compensate for this distortion between the pre- and intraoperative states. Some have attempted to recover the visible surface of the organ through image analysis and register it to the preoperative data, but this has proven insufficiently robust and may be problematic with large organs. A second approach is to introduce an intraoperative 3D imaging system as a transition. Hybrid operating rooms are becoming more and more popular, so this seems to be a viable solution, but current techniques require yet another external and constraining piece of apparatus such as an optical tracking system to determine the relationship between the intraoperative images and the endoscopic view. In this article, we propose a new approach to automatically register the reconstruction from an intraoperative CT acquisition with the static endoscopic view, by locating the endoscope tip in the volume data. We first describe our method to localize the endoscope orientation in the intraoperative image using standard image processing algorithms. Secondly, we highlight that the axis of the endoscope needs a specific calibration process to ensure proper registration accuracy. In the last section, we present quantitative and qualitative results proving the feasibility and the clinical potential of our approach.

A novel augmented reality to visualize the hidden organs and internal structure in surgeries

“…However, it does not offer depth perception of targeted area because of occlusion . Bernhardt et al proposed a system to automatic registration and reconstruction of CT data. This system gave the 0.1 mm in printing 3D model and registration errors below 15 pixels.…”

“…The methods used for this system were generic and thus did not consider the deformation factor of the organ. [17][18][19][20] Peterlík et al 21 proposed BMA to assists the surgeon's manoeuvre and accurate visual view of the surgical field (surgical field refers any part of the body in which the surgeons need to operate). This algorithm helps to reconstruct the 3D surface of body, and it improves the visualization of surgical field.…”

Marker-Based Registration for Large Deformations - Application to Open Liver Surgery

“…Attempts of AR have been realized in laparoscopy to improve the visualization of the screen, which appeared as an ideal interface between the numerical reconstruction and the surgeon [10], [11]. A 3D visualization and surgical planning software tools are proposed in [12] to overlay a 3D preoperative model on top of the stereoscopic camera of the Da Vinci robot; model registration is manually performed, but [13] proposed an automatic method to estimate the location of the endoscope by locating the tip of the volume data. Using only endoscopic cameras, Clements et al [14] proposes a rigid alignment approach based on salient anatomical features, extracted both in preoperative images and endoscopic data.…”

A novel enhanced intensity‐based automatic registration: Augmented reality for visualization and localization cancer tumors

“…Gillies et al 6 images, which will be of great significance to the proposed system. [8][9][10] Bernhardt et al 11 proposed a new automatic registration and reconstruction method by aligning the intraoperative CT images with the static endoscopic view. It used a 3D rotational C-arm and Zhang's method, which produced a registration accuracy of 1mm and processing time of 10-15s.…”

“…Green boxes specify the new proposed features and blue boxes specifies the existing features from Figure 2 Mahmoud et al 5It can search the object based on the given range of function instead of bidirectional search. These steps were taken to reduce reprojection error 11. Following completion of the optimization, we applied a transformation matrix to produce a rigid 3D transformation (aligned/registered images).…”