Tensor-based fusion of EEG and FMRI to understand neurological changes in schizophrenia

Abstract

Neuroimaging modalities such as functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) provide information about neurological functions in complementary spatiotemporal resolutions; therefore, fusion of these modalities is expected to provide better understanding of brain activity. In this paper, we jointly analyze fMRI and multi-channel EEG signals collected during an auditory oddball task with the goal of capturing brain activity patterns that differ between patients with schizophrenia and healthy controls. Rather than selecting a single electrode or matricizing the third-order tensor that can be naturally used to represent multi-channel EEG signals, we preserve the multi-way structure of EEG data and use a coupled matrix and tensor factorization (CMTF) model to jointly analyze fMRI and EEG signals. Our analysis reveals that (i) joint analysis of EEG and fMRI using a CMTF model can capture meaningful temporal and spatial signatures of patterns that behave differently in patients and controls, and (ii) these differences and the interpretability of the associated components increase by including multiple electrodes from frontal, motor and parietal areas, but not necessarily by including all electrodes in the analysis.

ACMTF for fusion of multi-modal neuroimaging data and identification of biomarkers

“…For this component, from the sMRI, we note that there is an increased concentration of gray matter throughout the parietal lobe for controls over patients and a decrease in concentration of gray matter in certain sections of the cerebellum for controls versus patients. This component is very similar to a component found using only the fMRI and EEG data [16]; however, we note that the structure of the default mode network (DMN) activation shown in the fMRI plot is much clearer when only the fMRI and EEG data are analyzed. The second component, displayed in Figure 3(b), whose EEG component describes the P2 and P3 peaks, shows higher fMRI activation in the superior parietal cortex and the visual cortex for the patients over the controls.…”

“…The second component, displayed in Figure 3(b), whose EEG component describes the P2 and P3 peaks, shows higher fMRI activation in the superior parietal cortex and the visual cortex for the patients over the controls. This component is nearly identical to a component found when only the fMRI and EEG data were analyzed [16]. However, from this joint analysis we find that this component is associated with an increase in concentration of gray matter in controls over patients in sections of the parietal lobe and cerebellum.…”

“…For the EEG data, for each electrode, we averaged small windows around the target tone across the repeated instances, deriving event-related potentials. Out of 64 electrodes in total, we used multiple electrodes from the frontal, motor and parietal areas, i.e., AF3, AF4, Fz, T7, C3, Cz, C4, T8, Pz, PO3 and PO, based on our previous results [16], which demonstrate better interpretability using this subset rather than all electrodes. Multi-channel EEG signals were then arranged as a third-order tensor: 32 subjects by 451 time samples by 11 electrodes.…”

“…Rather than such a CMTF model, which may fail to provide a unique solution in the presence of both shared/unshared components [15], we use a structure-revealing CMTF model, known as the advanced CMTF (ACMTF) model [15], revealing the weights of components in each modality to identify shared/unshared factors and understand the contribution from each modality. Previously, we have shown the promise of the ACMTF model in a joint analysis of fMRI and multi-channel EEG data in terms of capturing neural patterns that can differentiate patients with schizophrenia from healthy controls [16]. Our contributions in this paper are: (i) In addition to fMRI and EEG, we also incorporate sMRI data into the ACMTF model and demonstrate that the model can capture spatial and temporal patterns that differ between patients and controls.…”

Unraveling Diagnostic Biomarkers of Schizophrenia Through Structure-Revealing Fusion of Multi-Modal Neuroimaging Data

“…In this paper, we use a CMTF-based approach to jointly analyze neuroimaging signals from multiple modalities, more specifically, fMRI, sMRI and EEG data, collected during an auditory oddball (AOD) task from a group of subjects consisting of patients with schizophrenia and healthy controls with the goal of unraveling potential diagnostic biomarkers for schizophrenia. To the best of our knowledge, this is the first comprehensive study of a CMTF-based method for biomarker discovery for a psychiatric disorder discussing both strengths and limitations of the proposed framework, building onto our preliminary results in [35,36]. Furthermore, due to the reproducibility and uniqueness requirements of such an application, we use a structure-revealing CMTF model, known as the advanced CMTF (ACMTF) model [33], to estimate weights of the components in each modality in order to identify shared/unshared factors and quantify the contribution from each modality.…”