Photometric Stereo-Based Depth Map Reconstruction for Monocular Capsule Endoscopy

Yang, Hao; Li, Jing; Meng, Fei; Zhang, Peisen; Ciuti, Gastone; Dario, P.; Huang, Qiang

doi:10.3390/s20185403

Cited by 11 publications

(11 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another approach is to represent specular highlights using a mathematical parametric model. After modelling these highlights, depth as well as motion estimation can be performed from the specular highlight characteristics (Hao et al, 2020). However, this does not eliminate the problem of data loss and occluded information that is very important in applications such as polyp detection and feature matching.…”

Section: Endoscopic Specular Highlight Processingmentioning

confidence: 99%

“…One approach is to segment image regions where specularities are present and remove them from further analysis (Menor et al, 2016;Chikkerur et al, 2011), however, this may discard useful information from the images. Another approach is to mathematically model the physical properties of these highlights (Hao et al, 2020), which can help in estimating 3D shape and deformation of visualised anatomy. This however, is a very complex task that involves accurately modelling light reflection and scattering from the visualised tissue and properties of the camera light source.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Temporal Learning Approach to Inpainting Endoscopic Specularities and Its effect on Image Correspondence

Daher¹,

Vasconcelos²,

Stoyanov³

2022

Preprint

View full text Add to dashboard Cite

Video streams are utilised to guide minimally-invasive surgery and diagnostic procedures in a wide range of procedures, and many computer assisted techniques have been developed to automatically analyse them. These approaches can provide additional information to the surgeon such as lesion detection, instrument navigation, or anatomy 3D shape modeling. However, the necessary image features to recognise these patterns are not always reliably detected due to the presence of irregular light patterns such as specular highlight reflections. In this paper, we aim at removing specular highlights from endoscopic videos using machine learning. We propose using a temporal generative adversarial network (GAN) to inpaint the hidden anatomy under specularities, inferring its appearance spatially and from neighbouring frames where they are not present in the same location. This is achieved using in-vivo data of gastric endoscopy (Hyper-Kvasir) in a fully unsupervised manner that relies on automatic detection of specular highlights. System evaluations show significant improvements to traditional methods through direct comparison as well as other machine learning techniques through an ablation study that depicts the importance of the network's temporal and transfer learning components. The generalizability of our system to different surgical setups and procedures was also evaluated qualitatively on in-vivo data of gastric endoscopy and ex-vivo porcine data (SERV-CT, SCARED). We also assess the effect of our method in computer vision tasks that underpin 3D reconstruction and camera motion estimation, namely stereo disparity, optical flow, and sparse point feature matching. These are evaluated quantitatively and qualitatively and results show a positive effect of specular highlight inpainting on these tasks in a novel comprehensive analysis.

show abstract

Section: Endoscopic Specular Highlight Processingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A Temporal Learning Approach to Inpainting Endoscopic Specularities and Its effect on Image Correspondence

Daher¹,

Vasconcelos²,

Stoyanov³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The light source and camera of the endoscope are calibrated to obtain a model able to reproduce the photometry of the recordings. Based on recent work about photometric reconstruction 18 , we define a model that accounts for both the light emission and capture, as well as the interaction of the light with the geometry:…”

Section: Photometric Calibrationmentioning

confidence: 99%

EndoMapper dataset of complete calibrated endoscopy procedures

Azagra¹,

Sostres²,

Ferrández³

et al. 2022

Preprint

View full text Add to dashboard Cite

Computer-assisted systems are becoming broadly used in medicine. In endoscopy, most research focuses on automatic detection of polyps or other pathologies, but localization and navigation of the endoscope is completely performed manually by physicians. To broaden this research and bring spatial Artificial Intelligence to endoscopies, data from complete procedures are needed. This data will be used to build a 3D mapping and localization systems that can perform special task like, for example, detect blind zones during exploration, provide automatic polyp measurements, guide doctors to a polyp found in a previous exploration and retrieve previous images of the same area aligning them for easy comparison. These systems will provide an improvement in the quality and precision of the procedures while lowering the burden on the physicians. This paper introduces the Endomapper dataset, the first collection of complete endoscopy sequences acquired during regular medical practice, including slow and careful screening explorations, making secondary use of medical data. Its original purpose is to facilitate the development and evaluation of VSLAM (Visual Simultaneous Localization and Mapping) methods in real endoscopy data. The first release of the dataset is composed of 59 sequences with more than 15 hours of video. It is also the first endoscopic dataset that includes both the computed geometric and photometric endoscope calibration with the original calibration videos. Meta-data and annotations associated to the dataset varies from anatomical landmark and description of the procedure labeling, tools segmentation masks, COLMAP 3D reconstructions, simulated sequences with groundtruth and meta-data related to special cases, such as sequences from the same patient. This information will improve the research in endoscopic VSLAM, as well as other research lines, and create new research lines.

show abstract

“…In contrast, inside the human body, the illumination is controlled and light moves together with the camera. Recent work [13], [14] shows that changes in lighting, instead of being ignored, can be used to our benefit, obtaining dense reconstructions from monocular sequences.…”

Section: Related Workmentioning

confidence: 99%

Photometric single-view dense 3D reconstruction in endoscopy

Batlle

Montiel

Tardós

2022

2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Visual SLAM inside the human body will open the way to computer-assisted navigation in endoscopy. However, due to space limitations, medical endoscopes only provide monocular images, leading to systems lacking true scale. In this paper, we exploit the controlled lighting in colonoscopy to achieve the first in-vivo 3D reconstruction of the human colon using photometric stereo on a calibrated monocular endoscope. Our method works in a real medical environment, providing both a suitable in-place calibration procedure and a depth estimation technique adapted to the colon's tubular geometry. We validate our method on simulated colonoscopies, obtaining a mean error of 7% on depth estimation, which is below 3 mm on average. Our qualitative results on the EndoMapper dataset show that the method is able to correctly estimate the colon shape in real human colonoscopies, paving the ground for truescale monocular SLAM in endoscopy.

show abstract

Photometric Stereo-Based Depth Map Reconstruction for Monocular Capsule Endoscopy

Cited by 11 publications

References 41 publications

A Temporal Learning Approach to Inpainting Endoscopic Specularities and Its effect on Image Correspondence

A Temporal Learning Approach to Inpainting Endoscopic Specularities and Its effect on Image Correspondence

EndoMapper dataset of complete calibrated endoscopy procedures

Photometric single-view dense 3D reconstruction in endoscopy

Contact Info

Product

Resources

About