Model-Based Fusion of Multi-modal Volumetric Images: Application to Transcatheter Valve Procedures

Abstract

Abstract.Minimal invasive procedures such as transcatheter valve interventions are substituting conventional surgical techniques. Thus, novel operating rooms have been designed to augment traditional surgical equipment with advanced imaging systems to guide the procedures. We propose a novel method to fuse pre-operative and intra-operative information by jointly estimating anatomical models from multiple image modalities. Thereby high-quality patient-specific models are integrated into the imaging environment of operating rooms to guide cardiac interventions. Robust and fast machine learning techniques are utilized to guide the estimation process. Our method integrates both the redundant and complementary multimodal information to achieve a comprehensive modeling and simultaneously reduce the estimation uncertainty. Experiments performed on 28 patients with pairs of multimodal volumetric data are used to demonstrate high quality intra-operative patient-specific modeling of the aortic valve with a precision of 1.09mm in TEE and 1.73mm in 3D C-arm CT. Within a processing time of 10 seconds we additionally obtain model sensitive mapping between the pre-and intraoperative images.

A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions

“…Data from each modality are applied individually or combined with others. In 374 articles, im- age modality was present in 247, which were image data individually used in 129 articles [49], [51], [52], [54], [56], [57], [59]- [61], [63], [64] [103], [105], [106], [108], [110], [112]- [115], [117], [120]- [123], [128], [130], [134], [135], [137]- [149], [151], [152], [154], [155], [157], [158], [160], [162], [172]- [174], [176], [178], [191], [194]- [196], [198], [202], [203], [209], [210], [212], [214],…”

“…Fusion using the kernel approach means finding similarities between data points; for instance, kernel support vector machines (SVMs) are used to find the views in the data [45], [57], [106], [188]. Graphical models utilise the VOLUME 4, 2016 local and temporal structure of the modality, such as DBN [61] and dynamic Bayesian networks [62]. Neural network models are largely used in multimodal fusion [421].…”