Globally Optimal Fetoscopic Mosaicking Based on Pose Graph Optimisation With Affine Constraints

Abstract

Fetoscopic laser ablation surgery could be guided using a high-quality panorama of the operating site, representing a map of the placental vasculature. This can be achieved during the initial inspection phase of the procedure using image mosaicking techniques. Due to the lack of camera calibration in the operating room, it has been mostly modelled as an affine registration problem. While previous work mostly focuses on image feature extraction for visual odometry, the challenges related to large-scale reconstruction (re-localisation, loop closure, drift correction) remain largely unaddressed in this context. This letter proposes using pose graph optimisation to produce globally optimal image mosaics of placental vessels. Our approach follows the SLAM framework with a front-end for visual odometry and a back-end for long-term refinement. Our front-end uses a recent state-of-the-art odometry approach based on vessel segmentation, which is then managed by a key-frame structure and the bagof-words (BoW) scheme to retrieve loop closures. The backend, which is our key contribution, models odometry and loop closure constraints as a pose graph with affine warpings between states. This problem in the special Euclidean space cannot be solved by existing pose graph algorithms and available libraries such as G2O. We model states on affine Lie group with local linearisation in its Lie algebra. The cost function is established using Mahalanobis distance with the vectorisation of the Lie algebra. Finally, an iterative optimisation algorithm is adopted to minimise the cost function. The proposed pose graph optimisation is first validated on simulation data with a synthetic trajectory that has different levels of noise and different numbers of loop closures. Then the whole system is validated using real fetoscopic data that has three sequences with different numbers of frames and loop closures. Experimental results validate the advantage of the proposed method compared with baselines.

Robust fetoscopic mosaicking from deep learned flow fields

“…As future work, we plan to design a hybrid solution that switches between a vessel-based approach and optical flow depending on the most appropriate context and to develop reliable selection mechanisms. In addition, we plan to utilize methods for global optimization and reduction of error drift such as [ 17 ], to provide long-term consistency in large-scale fetoscopic mosaicking.…”

Anisotropic Generalized Bayesian Coherent Point Drift for Point Set Registration

“…In this part, different from previous methods, we assume q 3 as a Dirac delta function [37] which means that q 3 is only described by its mode. Therefore, we directly maximize the ELBO L(q) to derive the closed-form solution of q 3 instead of using the VBI's general solution (2). The variational optimization of this part can be regarded as the M-step of the EM algorithm.…”

“…P OINT set registration (PSR) is a fundamental research topic in robotics, computer vision [1], augmented reality (AR) and computer-assisted surgery (CAS) [2] [3]. The target of PSR is to recover the misalignment or the best transformation between two point sets (PS).…”

Qualitative Comparison of Image Stitching Algorithms for Multi-Camera Systems in Laparoscopy

“…The challenge is to create a seamless image without parallax artefacts whilst retaining important information and keeping the main objects in realistic positions. A very similar technique called image mosaicing is already widely used in laparoscopy (e.g., [ 11 , 12 , 13 , 14 , 15 ]). Instead of stitching images from different cameras in the same time frame, it creates a panorama from the consecutive frames of a unique moving camera.…”