Image-based navigation for functional endoscopic sinus surgery using structure from motion

Abstract

Functional Endoscopic Sinus Surgery (FESS) is a challenging procedure for otolaryngologists and is the main surgical approach for treating chronic sinusitis, to remove nasal polyps and open up passageways. To reach the source of the problem and to ultimately remove it, the surgeons must often remove several layers of cartilage and tissues. Often, the cartilage occludes or is within a few millimeters of critical anatomical structures such as nerves, arteries and ducts. To make FESS safer, surgeons use navigation systems that register a patient to his/her CT scan and track the position of the tools inside the patient. Current navigation systems, however, suffer from tracking errors greater than 1 mm, which is large when compared to the scale of the sinus cavities, and errors of this magnitude prevent from accurately overlaying virtual structures on the endoscope images. In this paper, we present a method to facilitate this task by 1) registering endoscopic images to CT data and 2) overlaying areas of interests on endoscope images to improve the safety of the procedure. First, our system uses structure from motion (SfM) to generate a small cloud of 3D points from a short video sequence. Then, it uses iterative closest point (ICP) algorithm to register the points to a 3D mesh that represents a section of a patients sinuses. The scale of the point cloud is approximated by measuring the magnitude of the endoscope’s motion during the sequence. We have recorded several video sequences from five patients and, given a reasonable initial registration estimate, our results demonstrate an average registration error of 1.21 mm when the endoscope is viewing erectile tissues and an average registration error of 0.91 mm when the endoscope is viewing non-erectile tissues. Our implementation SfM + ICP can execute in less than 7 seconds and can use as few as 15 frames (0.5 second of video). Future work will involve clinical validation of our results and strengthening the robustness to initial guesses and erectile tissues.

Repurposing Molecular Imaging and Sensing for Cancer Image–Guided Surgery

“…However, coregistration of the image field of view with patient anatomy using preoperative structural images is complicated by tissue deformation, and obtaining intraoperative structural images is challenging because of hardware complexity. To address these challenges, computational reconstruction of local 3-dimensional tissue structure has been developed for CT mapping (35). With the increasing adoption of minimally invasive surgery, there will be an increasing focus on this configuration for MIGS.…”

The deformable most-likely-point paradigm

“…A modified version of the learning-based photometric reconstruction technique developed by Reiter et al (2016) was used to reconstruct structures from endoscopic video collected from patients who volunteered to enroll in our study. Structure from motion (SfM) points obtained from video sequences (Leonard et al, 2016(Leonard et al, , 2018 were used to train a self-supervised deep neural network that enforces depth consistency between frames using relative pose information from SfM (Liu et al, 2018). This network was then used to predict the depth associated with each pixel in a single video frame.…”

Language-based translation and prediction of surgical navigation steps for endoscopic wayfinding assistance in minimally invasive surgery

“…Minimally invasive endoscopic surgery is valued as a standard in surgical practice, because with this method patient's New navigation systems employ an image-based approach for the registration of images and detection of objects to identify and use the image content of the endoscopic view for computer-assisted guidance in interventions [2]. Concerning the performance, the applied image processing algorithms are highly automatable and considerably reduce the need for tracking hardware and additional imaging information in the OR [3][4][5]. The virtualization trend has recently been intensified with deep learning applications that use neural networks to classify the observed anatomy and, thereby, the position of the endoscope [6,7].…”