Multi-view Emotion Recognition Using Deep Canonical Correlation Analysis

Jielin, Qiu,; Liu, Wei; Lu, Bao-Liang

doi:10.1007/978-3-030-04221-9_20

Cited by 42 publications

(23 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, minimizing loss is equivalent to maximizing correlation. The feature fusion layer is defined as the weighted average of the two transformed features [60]. Finally, the fused multimodal feature is fed into the support vector machine (SVM) to train the affective model.…”

Section: Deep Canonical Correlation Analysis Modelmentioning

confidence: 99%

Identifying Functional Brain Connectivity Patterns for EEG-Based Emotion Recognition

Zheng

2019

2019 9th International IEEE/EMBS Conference on Neural Engineering (NER)

Self Cite

View full text Add to dashboard Cite

Compared with the rich studies on the motor brain-computer interface (BCI), the recently emerging affective BCI presents distinct challenges since the brain functional connectivity networks involving emotion are not well investigated. Previous studies on emotion recognition based on electroencephalography (EEG) signals mainly rely on single-channel-based feature extraction methods. In this paper, we propose a novel emotion-relevant critical subnetwork selection algorithm and investigate three EEG functional connectivity network features: strength, clustering coefficient, and eigenvector centrality. The discrimination ability of the EEG connectivity features in emotion recognition is evaluated on three public emotion EEG datasets: SEED, SEED-V, and DEAP. The strength feature achieves the best classification performance and outperforms the state-of-the-art differential entropy feature based on single-channel analysis for the EEG signals. The experimental results reveal that distinct functional connectivity patterns are exhibited for the five emotions of disgust, fear, sadness, happiness, and neutrality. Furthermore, we construct a multimodal emotion recognition model by combining the functional connectivity features from EEG and the features from eye movements or physiological signals using deep canonical correlation analysis. The classification accuracies of multimodal emotion recognition are 95.08 ± 6.42% on the SEED dataset, 84.51 ± 5.11% on the SEED-V dataset, and 85.34 ± 2.90% and 86.61 ± 3.76% for arousal and valence on the DEAP dataset, respectively. The results demonstrate the complementary representation properties of the EEG functional connectivity network features with eye movement data. In addition, we find that the brain networks constructed with fewer channels, i.e., 18 channels, achieve comparable performance with that of the 62-channel network with respect to multimodal emotion recognition and enable easier setups for BCI systems in real scenarios.Index Terms-Affective brain-computer interface, EEG, eye movement, brain functional connectivity network, multimodal emotion recognition. ! •

show abstract

Section: Deep Canonical Correlation Analysis Modelmentioning

confidence: 99%

Identifying Functional Brain Connectivity Patterns for EEG-Based Emotion Recognition

Zheng

2019

2019 9th International IEEE/EMBS Conference on Neural Engineering (NER)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the experiment, since multi-modal features such as gaze features and visual features were used, this fusion method is considered to be suitable for comparison. Qiu et al proposed an emotional category classification method [44] by performing fusion of bio-information based on deep canonical correlation analysis (Deep CCA) [45]. Thus, we used the above state-of-the-art method as comparative method 4 (CM4) by using gaze features [42] and CNN features.…”

Section: Experimental Conditionsmentioning

confidence: 99%

Tensor-Based Emotional Category Classification via Visual Attention-Based Heterogeneous CNN Feature Fusion

Moroto

Maeda

Ogawa

et al. 2020

Sensors

View full text Add to dashboard Cite

The paper proposes a method of visual attention-based emotion classification through eye gaze analysis. Concretely, tensor-based emotional category classification via visual attention-based heterogeneous convolutional neural network (CNN) feature fusion is proposed. Based on the relationship between human emotions and changes in visual attention with time, the proposed method performs new gaze-based image representation that is suitable for reflecting the characteristics of the changes in visual attention with time. Furthermore, since emotions evoked in humans are closely related to objects in images, our method uses a CNN model to obtain CNN features that can represent their characteristics. For improving the representation ability to the emotional categories, we extract multiple CNN features from our novel gaze-based image representation and enable their fusion by constructing a novel tensor consisting of these CNN features. Thus, this tensor construction realizes the visual attention-based heterogeneous CNN feature fusion. This is the main contribution of this paper. Finally, by applying logistic tensor regression with general tensor discriminant analysis to the newly constructed tensor, the emotional category classification becomes feasible. Since experimental results show that the proposed method enables the emotional category classification with the F1-measure of approximately 0.6, and about 10% improvement can be realized compared to comparative methods including state-of-the-art methods, the effectiveness of the proposed method is verified.

show abstract

“…A recently developed deep canonical correlation analysis (DCCA) allows us to examine this possibility by approximating a non-linear mapping from neuronal ensemble activities to canonical variables with a DNN (Andrew et al, 2013). Previous non-invasive brain-computer interface (BCI) studies showed the effectiveness of DCCA as a means of feature extraction from electroencephalogram associated with various covariates of interest, such as eye movements and visual stimulus frequencies (Vu et al, 2016;Qiu et al, 2018;Liu et al, 2019). For example, Vu et al successfully improved the performance of the steady-state visual evoked potential-based BCI using DCCA-based feature extraction (Vu et al, 2016).…”

Section: Introductionmentioning

confidence: 99%

Decoding Kinematic Information From Primary Motor Cortex Ensemble Activities Using a Deep Canonical Correlation Analysis

2020

View full text Add to dashboard Cite

The control of arm movements through intracortical brain-machine interfaces (BMIs) mainly relies on the activities of the primary motor cortex (M1) neurons and mathematical models that decode their activities. Recent research on decoding process attempts to not only improve the performance but also simultaneously understand neural and behavioral relationships. In this study, we propose an efficient decoding algorithm using a deep canonical correlation analysis (DCCA), which maximizes correlations between canonical variables with the non-linear approximation of mappings from neuronal to canonical variables via deep learning. We investigate the effectiveness of using DCCA for finding a relationship between M1 activities and kinematic information when non-human primates performed a reaching task with one arm. Then, we examine whether using neural activity representations from DCCA improves the decoding performance through linear and non-linear decoders: a linear Kalman filter (LKF) and a long short-term memory in recurrent neural networks (LSTM-RNN). We found that neural representations of M1 activities estimated by DCCA resulted in more accurate decoding of velocity than those estimated by linear canonical correlation analysis, principal component analysis, factor analysis, and linear dynamical system. Decoding with DCCA yielded better performance than decoding the original FRs using LSTM-RNN (6.6 and 16.0% improvement on average for each velocity and position, respectively; Wilcoxon rank sum test, p < 0.05). Thus, DCCA can identify the kinematics-related canonical variables of M1 activities, thus improving the decoding performance. Our results may help advance the design of decoding models for intracortical BMIs.

show abstract

Multi-view Emotion Recognition Using Deep Canonical Correlation Analysis

Cited by 42 publications

References 12 publications

Identifying Functional Brain Connectivity Patterns for EEG-Based Emotion Recognition

Identifying Functional Brain Connectivity Patterns for EEG-Based Emotion Recognition

Tensor-Based Emotional Category Classification via Visual Attention-Based Heterogeneous CNN Feature Fusion

Decoding Kinematic Information From Primary Motor Cortex Ensemble Activities Using a Deep Canonical Correlation Analysis

Contact Info

Product

Resources

About