Multimodal Emotion Recognition Based on Ensemble Convolutional Neural Network

Abstract

In recent years, emotional recognition based on Electrophysiological (EEG) signals has become more and more popular. But the researchers ignored the fact that peripheral physiological signals can also reflect changes in mood. We propose an Ensemble Convolutional Neural Network (ECNN) model, which is used to automatically mine the correlation between multi-channel EEG signals and peripheral physiological signals in order to improve the emotion recognition accuracy. First, we design five convolution networks and use global average pooling (GAP) layers instead of fully connected layers; and then the plurality voting strategy is adopted to establish the ensemble model; eventually this model divides emotions into four categories. Based on the simulations on DEAP dataset, the experimental results demonstrate the superiority of the ECNN compared with other methods.

Multi-Feature Input Deep Forest for EEG-Based Emotion Recognition

“…For three-class emotion classification, Yao et al ( 2019 ) used deep forest with multi-scale window (MSWDF) to identify the emotions of pleasure, relaxation and sadness and achieved an accuracy of 84.90%. The proposed method achieved an accuracy of 76.8% for four emotions (shown in Figure 8 , which is higher than the ensemble convolutional neural network (ECNN) approach that obtained the accuracy of 73.76% (Huang et al, 2019 ). Besides, the current study takes five emotions for classification, including a neutral emotion, and achieved an average accuracy of 71.05%.…”

“…EEG-based emotion recognition is a hot-spot in recent years. Many researchers have proposed effective classification models to improve emotion recognition accuracy (Huang et al, 2019 ; Yao et al, 2019 ). The extraction of distinguish and consistent EEG features are critical for classification systems.…”

Multimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network

“…The second stage is the backward propagation and updating the weights. According to the error between the predicted value and the true value obtained by forward propagation, the error function of each network layer is obtained by backward propagation and the weights are updated [25]. This paper uses a layered incremental architecture within the CNN and constructs different network structures by setting different convolution kernel sizes and numbers in the convolutional layer.…”

“…Taking into account that the brain regions related to the frontal lobe have high recognition accuracy [28], the 6-channel EEG signals of the forehead and the PPS signals of other remaining channels are used as experimental data in the experiment. The data is downsampled to 128 Hz, and five bands including the delta (4-8 Hz), theta (8-13 Hz), alpha (13)(14)(15)(16)(17)(18)(19)(20)(21)(22)(23)(24)(25)(26)(27)(28)(29)(30), beta (30)(31)(32)(33)(34)(35)(36)(37)(38)(39)(40)(41)(42)(43), and gamma bands (4-43 Hz) are filtered out. Due to the error in the first 3 s of the video in the experiment, the first 3 s of the video are removed, and the middle 30 s of the remaining duration of the video are used as experimental data.…”

EEG based Emotion Recognition and Classification: a Review

“…Because of increasing demand of HCI and automatic human emotion recognition. Currently emotional research focuses more in the diagnosis of depression, mental illness, and assists health care professionals to make accurate diagnosis [3][4][5]. Human emotion can be recognize through physiological signal like Electromyography (EMG), Electrocardiography(ECG), Gelvonic skin response and Electroencephalography(EEG) from all of these, EEG is more effective because it provide more accurate, non-invasive and convenient way of capturing brain signal.…”