SLAM-based dense surface reconstruction in monocular Minimally Invasive Surgery and its application to Augmented Reality

Chen, Long; Tang, Wen; John, Nigel W.; Wan, Tao; Zhang, Jian Jun

doi:10.1016/j.cmpb.2018.02.006

Cited by 109 publications

(43 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The first approach, used in traditional laparoscopy, is based on moving a monocular endoscope in order to reconstruct the 3D surface of the surgical area. Three methods are commonly used to obtain depth information: Structure from Motion (SfM) [3,26], SLAM [27,28], and Shape from Shading (SfS) [29]. However, a disadvantage for both SfM and SLAM is that the camera needs to move constantly in order to obtain 3D information.…”

Section: D Reconstructionmentioning

confidence: 99%

Designing a New Endoscope for Panoramic-View with Focus-Area 3D-Vision in Minimally Invasive Surgery

et al. 2019

View full text Add to dashboard Cite

Purpose The minimally invasive surgery (MIS) has shown advantages when compared to traditional surgery. However, there are two major challenges in the MIS technique: the limited field of view (FOV) and the lack of depth perception provided by the standard monocular endoscope. Therefore, in this study, we proposed a New Endoscope for Panoramic-View with Focus-Area 3D-Vision (3DMISPE) in order to provide surgeons with a broad view field and 3D images in the surgical area for real-time display. Method The proposed system consisted of two endoscopic cameras fixed to each other. Compared to our previous study, the proposed algorithm for the stitching videos was novel. This proposed stitching algorithm was based on the stereo vision synthesis theory. Thus, this new method can support 3D reconstruction and image stitching at the same time. Moreover, our approach employed the same functions on reconstructing 3D surface images by calculating the overlap region's disparity and performing image stitching with the two-view images from both the cameras. Results The experimental results demonstrated that the proposed method can combine two endoscope's FOV into one wider FOV. In addition, the part in the overlap region could also be synthesized for a 3D display to provide more information about depth and distance, with an error of about 1 mm. In the proposed system, the performance could achieve a frame rate of up to 11.3 fps on a single Intel i5-4590 CPU computer and 17.6 fps on a computer with an additional GTX1060 Nvidia GeForce GPU. Furthermore, the proposed stitching method in this study could be made 1.4 times after when compared to that in our previous report. Besides, our method also improved stitched image quality by significantly reducing the alignment errors or "ghosting" when compared to the SURF-based stitching method employed in our previous study. Conclusion The proposed system can provide a more efficient way for the doctors with a broad area of view while still providing a 3D surface image in real-time applications. Our system give promises to improve existing limitations in laparoscopic surgery such as the limited FOV and the lack of depth perception.

show abstract

Section: D Reconstructionmentioning

confidence: 99%

Designing a New Endoscope for Panoramic-View with Focus-Area 3D-Vision in Minimally Invasive Surgery

et al. 2019

View full text Add to dashboard Cite

show abstract

“…15 In a hospital, all staff of the hospital share all data from all patients on the server, which has to control as a multi-leveled security following the data state. 13 All patients' leveled data will be stored after finishing an encrypting algorithm;…”

Section: Secure Structurementioning

confidence: 99%

“…12 Chen et al presented a novel intra-operative dense surface reconstruction framework that is capable of providing geometry information from only monocular MIS videos for geometry-aware AR applications such as site measurements and depth cues using SLAM-based Dense Surface Reconstruction. 13 Medical data recognition is another FIGURE 1 Augmented reality system important challenge. These include some classical algorithms like data processing, analysis procedures, pattern classification, neural modeling, and genetic computation.…”

Section: Introductionmentioning

confidence: 99%

Augmented reality for botulinum toxin injection

Kim

Jeong

Seo

et al. 2019

Concurrency and Computation

View full text Add to dashboard Cite

Summary Augmented‐reality (AR) devices allow physicians to incorporate data visualization into diagnostic and treatment procedures to improve work efficiency and safety and reduce cost. They are also used to enhance surgical training. In this study, we implemented an AR application for Botox injections using a face recognition algorithm based on deep learning, and we evaluated the recognition accuracy of this application using 27 participants. The accuracy was around 3 mm for all parts of the facial region. The method of increasing surgical efficiency with AR is accurate enough to be used for surgery and provides great potential for further development.

show abstract

“…It may cause the visualization error and it is essential for the surgery to visualize the tumors, blood vessels, nerves etc . Chen et al explored a different approach to provide accurate camera tracking in visualizing the kidney in endoscopic surgery. It uses Simultaneous Localization and Mapping (SLAM) algorithm.…”

Section: Introductionmentioning

confidence: 99%

“…It may cause the visualization error and it is essential for the surgery to visualize the tumors, blood vessels, nerves etc. 12 Chen et al 13 The motive of this research is develop an Automatic registration of AR-based surgery system to meet the high registration accuracy and low computational cost. In order to improve the registration accuracy and processing time we are focused on the removal of manual processes while registration process (selecting landmarks).…”

Section: Introductionmentioning

confidence: 99%

A novel enhanced intensity‐based automatic registration: Augmented reality for visualization and localization cancer tumors

Tan

Alsadoon

Prasad

et al. 2020

Robotics Computer Surgery

View full text Add to dashboard Cite

The purpose of this study is to replace the manual process (selecting the landmarks on mesh and anchor points on the video) by Intensity‐based Automatic Registration method to reach registration accuracy and low processing time. The proposed system consists of an Enhanced Intensity‐based Automatic Registration (EIbAR) using Modified Zero Normalized Cross Correlation (MZNCC) algorithm. The proposed system was implemented on videos of breast cancer tumors. Results showed that the proposed algorithm—as compared to a reference—improved registration accuracy by an average of 2 mm. In addition, the proposed algorithm—as compared to a reference—reduced the number of pixel matching, thereby reducing processing time on the video by an average of 22 ms/frame. The proposed system can, thus, provide an acceptable accuracy and processing time during scene augmentation of videos, which provides a seamless use of augmented‐reality for surgeons in visualizing cancer tumors.

show abstract

SLAM-based dense surface reconstruction in monocular Minimally Invasive Surgery and its application to Augmented Reality

Cited by 109 publications

References 50 publications

Designing a New Endoscope for Panoramic-View with Focus-Area 3D-Vision in Minimally Invasive Surgery

Designing a New Endoscope for Panoramic-View with Focus-Area 3D-Vision in Minimally Invasive Surgery

Augmented reality for botulinum toxin injection

A novel enhanced intensity‐based automatic registration: Augmented reality for visualization and localization cancer tumors

Contact Info

Product

Resources

About