Real-time mosaicing of fetoscopic videos using SIFT

Abstract

Fetoscopic laser photo-coagulation of the placental vascular anastomoses remains the most effective therapy for twin-to-twin transfusion syndrome (TTTS) in monochorionic twin pregnancies. However, to ensure the success of the intervention, complete photo-coagulation of all anastomoses is needed. This is made difficult by the limited field of view of the fetoscopic video guidance, which hinders the surgeon's ability to locate all the anastomoses. A potential solution to this problem is to expand the field of view of the placental surface by creating a mosaic from overlapping fetoscopic images. This mosaic can then be used for anastomoses localization and spatial orientation during surgery. However, this requires accurate and fast algorithms that can operate within the real-time constraints of fetal surgery. In this work, we present an image mosaicing framework that leverages the parallelism of modern GPUs and can process clinical fetoscopic images in real-time. Initial qualitative results on ex-vivo placental images indicate that the proposed framework can generate clinically useful mosaics from fetoscopic videos in real-time. DESCRIPTION OF PURPOSETwin-to-twin transfusion syndrome (TTTS) is a complication that affects 9% of monochorionic identical twin pregnancies, where two or more fetuses share a common placenta. 1 The disease results from an imbalance in the blood circulation between the twins due to the presence of anastomoses in the monochorionic placenta. The implication of TTTS is that one of the twins (the donor ) does not receive an adequate supply of blood for normal growth, while the other twin (the recipient) receives too much blood resulting in an overloaded cardiovascular system. Without intervention, the condition is often fatal for both twins. In advanced stages of TTTS, laser coagulation of the connecting vessels on the placenta between the twin fetuses can be a curative procedure. Under fetoscopic video guidance, the surgeon uses a laser fibre to photo-coagulate the blood vessels that connect the two fetuses. This involves visually inspecting the entire placenta to identify the crossing blood vessels that contribute to placental anastomoses. Laser energy is then used to coagulate these blood vessels, effectively separating the shared placenta and allowing each twin to develop independently.A key limitation of fetoscopic video guidance is the small field of view, which makes it difficult for the surgeon to visualize the full placental structure and ensure that all of the vascular anastomoses are identified and treated (figure 1). This problem may be overcome by creating a larger field of view through mosaicing of the fetoscopic video frames. The placental image mosaic may be helpful in locating the anastomoses and for spatial orientation during surgery. 2 These endoscopic image mosaics could also be fused with other imaging modalities to augment the information available during surgery. 3,4 The key to generating an accurate and usable image mosaic is to be able to extract accurate correspondences...

Refractive Structure-from-Motion Through a Flat Refractive Interface

“…A perspective work will be to integrate the proposed approach within underwater mosaicking [7] or Refractive Simultaneous Localization And Mapping pipeline. This will require to develop robust underwater registration methods adapted to intended applicative context (e.g learning-based approaches) or combine it with robotic imaging [8].…”

“…The interface to camera centre distance along the camera's axis is denoted as d. An image point i observed in view j is denoted p [2,2], [2,4], [3,5], [4,3], [4,7], [5,6], [5,8], [6,9]) = 1.…”

Technical note: probabilistic visual and electromagnetic data fusion for robust drift-free sequential mosaicking. Application to fetoscopy

“…4. We assume that the camera is moving smoothly and therefore model the relation between camera poses with a constant velocity motion model 18 with mean µ p (x k 1 , x k 2 ) and diagonal covariance matrix ⌃ p .…”

GIFT-Grab: Real-time C++ and Python multi-channel video capture, processing and encoding API

“…Name: Zenodo [10,9]. Other examples include real-time stereo reconstruction [48] and probe tracking [49] in robotic surgery, surveillance endoscopies [50], real-time panorama image synthesis [51], vehicle surveillance [52] and content-based video identification [53].…”

“…GIFT-Surg involves major innovations in science and engineering combined with clinical translation for improving fetal therapy and diagnosis by providing advanced image processing and visualisation capabilities. We have already obtained promising results with a number of novel methods including real-time mosaicing of the placenta [9] and sensor-free real-time instrument tracking [10]. These methods rely on real-time video streams from medical devices such as endoscopes and ultrasound probes.…”