Cost-Efficient Video Synthesis and Evaluation for Development of Virtual 3D Endoscopy

Abstract

Objective: 3D reconstruction of the shape and texture of hollow organs captured by endoscopy is important for the diagnosis and surveillance of early and recurrent cancers. Better evaluation of 3D reconstruction pipelines developed for such applications requires easy access to extensive datasets and associated ground truths, cost-efficient and scalable simulations of a range of possible clinical scenarios, and more reliable and insightful metrics to assess performance. Methods: We present a computer-aided simulation platform for cost-effective synthesis of monocular endoscope videos and corresponding ground truths that mimic a range of potential settings and situations one might encounter during acquisition of clinical endoscopy videos. Using cystoscopy of the bladder as model case, we generated an extensive dataset comprising several synthesized videos of a bladder phantom. We then introduce a novel evaluation procedure to reliably assess an individual 3D reconstruction pipeline or to compare different pipelines. Results: To illustrate the use of the proposed platform and evaluation procedure, we use the aforementioned dataset and ground truths to evaluate a proprietary 3D reconstruction pipeline (CYSTO3D) for bladder cystoscopy videos and compared it with a general-purpose 3D reconstruction pipeline (COLMAP). The evaluation results provide insight into the suggested clinical acquisition protocol and several potential areas for refinement of the pipeline to improve future performance. Conclusion: Our work proposes an endoscope video synthesis and reconstruction evaluation toolset and presents experimental results that illustrate usage of the toolset to efficiently assess performance and reveal possible problems of any given 3D reconstruction pipeline, to compare different pipelines, and to provide technically or clinically actionable insights.

Towards improved 3D reconstruction of cystoscopies through real-time feedback for frame reacquisition

“…The following metrics were considered for use in the feature vector, mostly due to their ability to detect blur or sudden changes in the frame usefulness, such as due to the presence of bladder debris: Feature count: In our analysis of past work, we noted that a common reason for failure in 3D reconstructions was a lack of salient scale invariant feature transform (SIFT) features to enable frame matching in the SfM step [ 5 , 17 ]. Thus, we hypothesized that an insufficient number of features would limit frame utility.…”

“…Feature count: In our analysis of past work, we noted that a common reason for failure in 3D reconstructions was a lack of salient scale invariant feature transform (SIFT) features to enable frame matching in the SfM step [ 5 , 17 ]. Thus, we hypothesized that an insufficient number of features would limit frame utility.…”

Fiberscopic Pattern Removal for Optimal Coverage in 3D Bladder Reconstructions of Fiberscope Cystoscopy Videos

“…Our goal was to design a pre-processing algorithm for robust 3D reconstruction of fiberscope videos with CYSTO3D. It was thus important to consider methods that would preserve contrast and minimize blur, as loss of contrast and blurring are problematic for reconstruction [29]. The algorithm comprises three general steps (Fig.…”

“…We also introduce a novel metric to assess the completeness of a reconstruction, which we term the area of reconstruction coverage. This metric differs from other metrics of reconstruction completeness in that it does not require ground truth knowledge about the surface area of the scene being reconstructed [28,29].…”

In silico simulation: a key enabling technology for next-generation intelligent surgical systems

“…In this case, in silico training enables them to generate scalar ground truth masks for the amount of smoke, decomposing the learning problem into a two-step process and enabling smoke removal via generative cooperative networks. Zhou et al [127] develop an open-source simulation tool based on Blender [80] for data generation of synthetic endoscopic videos and relevant ground truths that can allow vigorous evaluation of 3D reconstruction methods.…”