Reconstructing Sinus Anatomy from Endoscopic Video – Towards a Radiation-Free Approach for Quantitative Longitudinal Assessment

Liu, Xingtong; Stiber, Maia; Huang, Jason; Ishii, Masaru; Hager, Gregory D.; Taylor, Russell H.; Unberath, Mathias

doi:10.1007/978-3-030-59716-0_1

Cited by 27 publications

(8 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By fusing consecutive depth maps and simultaneously estimating the endoscope motion using geometric constraints, it has been demonstrated that long range colon sections could be reconstructed [46]. A similar approach has also been successfully applied to 3-D reconstruction of the sinus anatomy from endoscopic video so as to propose an alternative to CT scans – expensive procedures using ionizing radiation – for longitudinal monitoring of patients after nasal obstruction surgery [47]. However, critical limitations, such as navigation within deformable environments, need to be overcome.…”

Section: Computer-assisted Navigationmentioning

confidence: 99%

Computer Vision in the Surgical Operating Room

et al. 2020

View full text Add to dashboard Cite

Background: Multiple types of surgical cameras are used in modern surgical practice and provide a rich visual signal that is used by surgeons to visualize the clinical site and make clinical decisions. This signal can also be used by artificial intelligence (AI) methods to provide support in identifying instruments, structures, or activities both in real-time during procedures and postoperatively for analytics and understanding of surgical processes. Summary: In this paper, we provide a succinct perspective on the use of AI and especially computer vision to power solutions for the surgical operating room (OR). The synergy between data availability and technical advances in computational power and AI methodology has led to rapid developments in the field and promising advances. Key Messages: With the increasing availability of surgical video sources and the convergence of technologiesaround video storage, processing, and understanding, we believe clinical solutions and products leveraging vision are going to become an important component of modern surgical capabilities. However, both technical and clinical challenges remain to be overcome to efficiently make use of vision-based approaches into the clinic.

show abstract

Section: Computer-assisted Navigationmentioning

confidence: 99%

Computer Vision in the Surgical Operating Room

et al. 2020

View full text Add to dashboard Cite

show abstract

“…When the depth predictions of frames were placed together to generate a 3D reconstruction mesh, their shapes were not aligned, causing a sparse and broken surface. To compensate, the authors introduced an additional averaging step and adjustment of the depth using SfM [1,22]. However, these post-processing steps prevented the programs' real-time execution.…”

Section: … Alert!!! Unsurveyed Regionmentioning

confidence: 99%

“…However, their depth estimation network can only handle simple cases and is vulnerable to environmental noise, and the predicted shapes often fail to produce good quality meshes. Later work [1,22] exploited the possibility to integrate SfM with the learning-based depth estimation to calibrate depth predictions, but the time expense brought in by SfM restricts the methods from large-scale reconstruction applications.…”

Section: D Reconstruction In Endoscopymentioning

confidence: 99%

ColDE: A Depth Estimation Framework for Colonoscopy Reconstruction

Zhang

Frahm

Ehrenstein

et al. 2021

Preprint

View full text Add to dashboard Cite

One of the key elements of reconstructing a 3D mesh from a monocular video is generating every frame's depth map. However, in the application of colonoscopy video reconstruction, producing good-quality depth estimation is challenging. Neural networks can be easily fooled by photometric distractions or fail to capture the complex shape of the colon surface, predicting defective shapes that result in broken meshes. Aiming to fundamentally improve the depth estimation quality for colonoscopy 3D reconstruction, in this work we have designed a set of training losses to deal with the special challenges of colonoscopy data. For better training, a set of geometric consistency objectives was developed, using both depth and surface normal information. Also, the classic photometric loss was extended with feature matching to compensate for illumination noise. With the training losses powerful enough, our self-supervised framework named ColDE is able to produce better depth maps of colonoscopy data as compared to the previous work utilizing prior depth knowledge. Used in reconstruction, our network is able to reconstruct good-quality colon meshes in real-time without any post-processing, making it the first to be clinically applicable.

show abstract

“…Lastly, the changes of camera view points compound the aforementioned difficulties into an ego motion task with dynamic objects. Even though there exists prior works in 3D reconstruction in surgical scene, they are generally limited by assuming a static scene [12] or no presence of surgical tools [20].…”

Section: Introductionmentioning

confidence: 99%

E-DSSR: Efficient Dynamic Surgical Scene Reconstruction with Transformer-based Stereoscopic Depth Perception

Long

Yee

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Reconstructing the scene of robotic surgery from the stereo endoscopic video is an important and promising topic in surgical data science, which potentially supports many applications such as surgical visual perception, robotic surgery education and intra-operative context awareness. However, current methods are mostly restricted to reconstructing static anatomy assuming no tissue deformation, tool occlusion and de-occlusion, and camera movement. However, these assumptions are not always satisfied in minimal invasive robotic surgeries. In this work, we present an efficient reconstruction pipeline for highly dynamic surgical scenes that runs at 28 fps. Specifically, we design a transformer-based stereoscopic depth perception for efficient depth estimation and a lightweight tool segmentor to handle tool occlusion. After that, a dynamic reconstruction algorithm which can estimate the tissue deformation and camera movement, and aggregate the information over time is proposed for surgical scene reconstruction. We evaluate the proposed pipeline on two datasets, the public Hamlyn Centre Endoscopic Video Dataset and our in-house DaVinci robotic surgery dataset. The results demonstrate that our method can recover the scene obstructed by the surgical tool and handle the movement of camera in realistic surgical scenarios effectively at real-time speed.

show abstract

Reconstructing Sinus Anatomy from Endoscopic Video – Towards a Radiation-Free Approach for Quantitative Longitudinal Assessment

Cited by 27 publications

References 23 publications

Computer Vision in the Surgical Operating Room

Computer Vision in the Surgical Operating Room

ColDE: A Depth Estimation Framework for Colonoscopy Reconstruction

E-DSSR: Efficient Dynamic Surgical Scene Reconstruction with Transformer-based Stereoscopic Depth Perception

Contact Info

Product

Resources

About