Deep learning-based fetoscopic mosaicking for field-of-view expansion

Bano, Sophia; Vasconcelos, Francisco; Tella-Amo, Marcel; Dwyer, George; Gruijthuijsen, Caspar; Poorten, Emmanuel Vander; Vercauteren, Tom; Ourselin, Sébastien; Deprest, Jan; Stoyanov, Danail

doi:10.1007/s11548-020-02242-8

Cited by 28 publications

(43 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Beyond automatic diagnosis and super‐resolution approaches in IVM, recent advances also highlight ways in which DL can enable novel instrumentation development and image reconstructions to enable new functionalities for compact microscopy systems. Such examples include multispectral endomicroscopy [129], more robust mosaicking for FOV expansion [130], and end‐to‐end image reconstruction using disordered fiber‐optic probes [131,132]. We anticipate that similarly to ex vivo microscopy, in the coming years DL will be increasingly utilized to overcome physical constraints, augment contrast mechanisms, and enable new capabilities for IVM systems.…”

Section: Applications In Biomedical Opticsmentioning

confidence: 99%

Deep Learning in Biomedical Optics

et al. 2021

View full text Add to dashboard Cite

This article reviews deep learning applications in biomedical optics with a particular emphasis on image formation. The review is organized by imaging domains within biomedical optics and includes microscopy, fluorescence lifetime imaging, in vivo microscopy, widefield endoscopy, optical coherence tomography, photoacoustic imaging, diffuse tomography, and functional optical brain imaging. For each of these domains, we summarize how deep learning has been applied and highlight methods by which deep learning can enable new capabilities for optics in medicine. Challenges and opportunities to improve translation and adoption of deep learning in biomedical optics are also summarized. Lasers Surg. Med. © 2021 Wiley Periodicals LLC

show abstract

Section: Applications In Biomedical Opticsmentioning

confidence: 99%

Deep Learning in Biomedical Optics

et al. 2021

View full text Add to dashboard Cite

show abstract

“…In this setting, most breakthroughs have been achieved with direct methods. This includes pixel-wise gradient alignment [18], a deep learning approach for direct homography regression [19], [20], and more recently, registration of segmented placental vessels [13]. While this last method shows significant progress, it ignores accumulative drift errors that would eventually occur in long videos.…”

Section: Related Workmentioning

confidence: 99%

Globally Optimal Fetoscopic Mosaicking Based on Pose Graph Optimisation With Affine Constraints

Bano

Deprest

et al. 2021

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

Fetoscopic laser ablation surgery could be guided using a high-quality panorama of the operating site, representing a map of the placental vasculature. This can be achieved during the initial inspection phase of the procedure using image mosaicking techniques. Due to the lack of camera calibration in the operating room, it has been mostly modelled as an affine registration problem. While previous work mostly focuses on image feature extraction for visual odometry, the challenges related to large-scale reconstruction (re-localisation, loop closure, drift correction) remain largely unaddressed in this context. This letter proposes using pose graph optimisation to produce globally optimal image mosaics of placental vessels. Our approach follows the SLAM framework with a front-end for visual odometry and a back-end for long-term refinement. Our front-end uses a recent state-of-the-art odometry approach based on vessel segmentation, which is then managed by a key-frame structure and the bagof-words (BoW) scheme to retrieve loop closures. The backend, which is our key contribution, models odometry and loop closure constraints as a pose graph with affine warpings between states. This problem in the special Euclidean space cannot be solved by existing pose graph algorithms and available libraries such as G2O. We model states on affine Lie group with local linearisation in its Lie algebra. The cost function is established using Mahalanobis distance with the vectorisation of the Lie algebra. Finally, an iterative optimisation algorithm is adopted to minimise the cost function. The proposed pose graph optimisation is first validated on simulation data with a synthetic trajectory that has different levels of noise and different numbers of loop closures. Then the whole system is validated using real fetoscopic data that has three sequences with different numbers of frames and loop closures. Experimental results validate the advantage of the proposed method compared with baselines.

show abstract

“…Large intra-case variation can also be observed from these representative images. All these factors contribute towards limiting the performance of the existing placental image segmentation and registration methods [19,18,27].…”

Section: Segmentation Dataset Descriptionmentioning

confidence: 99%

FetReg: Placental Vessel Segmentation and Registration in Fetoscopy Challenge Dataset

Bano¹,

Casella²,

Vasconcelos³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Fetoscopy laser photocoagulation is a widely used procedure for the treatment of Twin-to-Twin Transfusion Syndrome (TTTS), that occur in mono-chorionic multiple pregnancies due to placental vascular anastomoses. This procedure is particularly challenging due to limited field of view, poor manoeuvrability of the fetoscope, poor visibility due to fluid turbidity, variability in light source, and unusual position of the placenta. This may lead to increased procedural time and incomplete ablation, resulting in persistent TTTS. Computer-assisted intervention may help overcome these challenges by expanding the fetoscopic field of view through video mosaicking and providing better visualization of the vessel network. However, the research and development in this domain remain limited due to unavailability of high-quality data to encode the intra-and inter-procedure variability. Through the Fetoscopic Placental Vessel Segmentation and Registration (FetReg) challenge, we present a large-scale multi-centre dataset for the development of generalized and robust semantic segmentation and video mosaicking algorithms for the fetal environment with a focus on creating drift-free mosaics from long duration fetoscopy videos. In this paper, we provide an overview of the FetReg dataset, challenge tasks, evaluation metrics and baseline methods for both segmentation and registration. Baseline methods results on the FetReg dataset shows that our dataset poses interesting challenges, offering large opportunity for the creation of novel methods and models through a community effort initiative guided by the FetReg challenge.

show abstract

Deep learning-based fetoscopic mosaicking for field-of-view expansion

Cited by 28 publications

References 22 publications

Deep Learning in Biomedical Optics

Deep Learning in Biomedical Optics

Globally Optimal Fetoscopic Mosaicking Based on Pose Graph Optimisation With Affine Constraints

FetReg: Placental Vessel Segmentation and Registration in Fetoscopy Challenge Dataset

Contact Info

Product

Resources

About