Real‐time geometry‐aware augmented reality in minimally invasive surgery

Abstract

The potential of augmented reality (AR) technology to assist minimally invasive surgery (MIS) lies in its computational performance and accuracy in dealing with challenging MIS scenes. Even with the latest hardware and software technologies, achieving both real-time and accurate augmented information overlay in MIS is still a formidable task. In this Letter, the authors present a novel real-time AR framework for MIS that achieves interactive geometric aware AR in endoscopic surgery with stereo views. The authors’ framework tracks the movement of the endoscopic camera and simultaneously reconstructs a dense geometric mesh of the MIS scene. The movement of the camera is predicted by minimising the re-projection error to achieve a fast tracking performance, while the three-dimensional mesh is incrementally built by a dense zero mean normalised cross-correlation stereo-matching method to improve the accuracy of the surface reconstruction. The proposed system does not require any prior template or pre-operative scan and can infer the geometric information intra-operatively in real time. With the geometric information available, the proposed AR framework is able to interactively add annotations, localisation of tumours and vessels, and measurement labelling with greater precision and accuracy compared with the state-of-the-art approaches.

Planning and marking for thoracoscopic anatomical segmentectomies

“…Finally, the interactive augmented reality (RAI) initially developed for liver surgery is slowly progressing (34,47). This appealing technique comes up against unresolved technical constraints in thoracic surgery to date, i.e., deflation of the lung.…”

Recent Developments and Future Challenges in Medical Mixed Reality

“…In this way, manual alignment or fiducial markers were not required and tissue deformation caused by insufflation and respiration during MIS was taken into account. Chen et al [110] presented a geometry-aware AR environment in MIS by combining SLAM algorithm with dense stereo reconstruction to provide a global surface mesh for intra-operative AR interactions such as area highlighting and measurement. (see Figure 7).…”

A novel enhanced intensity‐based automatic registration: Augmented reality for visualization and localization cancer tumors

“…Next, we computed for the dissimilarity between the pre‐operative and intra‐operative images using MZNCC. This step involved the use of a Zero Normalized Cross Correlation instead of Normalized Cross Correlation along with the winner takes all strategy. ZNCC is used to find the dissimilarity measure between the two images.…”

“…Regardless, the combination of algorithms and methods can be of importance to the proposed system. Chen et al investigated on the accuracy of surface reconstruction to visualize livers in minimally invasive surgery using Zero Normalized Cross Correlation (ZNCC) algorithm. This system achieved an accuracy of 2.37 mm which is slightly better than the existing system.…”