Multimodal Data Fusion Using Source Separation: Application to Medical Imaging

Abstract

The Joint ICA (jICA) and the Transposed IVA (tIVA) models are two effective solutions based on blind source separation that enable fusion of data from multiple modalities in a symmetric and fully multivariate manner. In [1], their properties and the major issues in their implementation are discussed in detail. In this accompanying paper, we consider the application of these two models to fusion of multi-modal medical imaging data-functional magnetic resonance imaging (fMRI), structural MRI (sMRI), and electroencephalography (EEG) data collected from a group of healthy controls and patients with schizophrenia performing an auditory oddball task. We show how both models can be used to identify a set of components that report on differences between the two groups, jointly, for all the modalities used in the study. We discuss the importance of algorithm and order selection as well as trade-offs involved in the selection of one model over another. We note that for the selected dataset, especially given the limited number of subjects available for the study, jICA provides a more desirable solution, however the use of an ICA algorithm that uses flexible density matching provides advantages over the most widely used algorithm, Infomax, for the problem.

ACMTF for fusion of multi-modal neuroimaging data and identification of biomarkers

“…Therefore, joint analyses of signals from multiple neuroimaging modalities are of great interest to better understand neurological disorders [2], [3]. Functional modalities such as electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) can be used to study the changes in neural activities triggered by an event in both patients affected by schizophrenia as well as healthy controls [2].…”

“…Therefore, joint analyses of signals from multiple neuroimaging modalities are of great interest to better understand neurological disorders [2], [3]. Functional modalities such as electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) can be used to study the changes in neural activities triggered by an event in both patients affected by schizophrenia as well as healthy controls [2]. In addition to functional methods, anatomical neuroimaging techniques such as structural MRI (sMRI) can also capture structural differences between patients and controls [1], and since structure underlies function, joint analysis of these three modalities is expected to provide a more comprehensive picture.…”

“…One type of heterogeneity is the order of the data, e.g., while multi-channel EEG signals can be represented as a third-order tensor with modes: subjects, time samples and electrodes, fMRI and sMRI signals are often arranged as subjects by voxels matrices (Figure 1). The traditional approach used for jointly analyzing such neuroimaging data, first either matricizes the higher-order tensors [4] or selects a single EEG electrode [2], thus ignoring the potential multilinear structures of multi-channel EEG signals. Then these approaches use matrix factorization-based fusion methods such as joint independent component analysis (jICA) [5], linked ICA [6], parallel ICA [7] and transposed independent vector analysis (tIVA) [2].…”

“…The traditional approach used for jointly analyzing such neuroimaging data, first either matricizes the higher-order tensors [4] or selects a single EEG electrode [2], thus ignoring the potential multilinear structures of multi-channel EEG signals. Then these approaches use matrix factorization-based fusion methods such as joint independent component analysis (jICA) [5], linked ICA [6], parallel ICA [7] and transposed independent vector analysis (tIVA) [2]. However, matrix factorization-based fusion models need additional constraints such as orthogonality and independence to obtain a unique model, and may fail to capture the true factors due to those constraints [8].…”

Application of Morphological Component Analysis to Optical Image Fusion

“…They help greatly to make a decision. The application fields of the data fusion are varied and diverse: Medical imaging (Magtibay et al, 2016;Adali et al, 2015), economy (Barenboim and Pearl, 2016), information theory (Gagolewski, 2016), image processing (Paris and Bruzzone, 2015), etc. In remote sensing where the nature and the resolution of sensors are various and different, methods of image fusion implement several types of images: Panchromatic (PAN), Multispectral (MS), Hyperspectral (HS) and Radar (SAR).…”