CNN and LSTM-Based Emotion Charting Using Physiological Signals

Dar, Muhammad Najam; Akram, M. Usman; Khawaja, Sajid Gul; Pujari, Amit N.

doi:10.3390/s20164551

Cited by 85 publications

(35 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Combining the benefits of LSTM and CNN, a deep-learning architecture is proposed by Dar et al 6 , which has proved to be effective at recognizing emotions and outperforming previous approaches. Their method is subject-agnostic, combining two publicly available data sets (DREAMER and AMIGOS) with low-cost, wearable sensors to extract physiological signals suitable for real-world situations.…”

Section: Related Workmentioning

confidence: 99%

CNN-XGBoost fusion-based affective state recognition using EEG spectrogram image analysis

Khan

Salsabil

Alam

et al. 2022

Sci Rep

View full text Add to dashboard Cite

Recognizing emotional state of human using brain signal is an active research domain with several open challenges. In this research, we propose a signal spectrogram image based CNN-XGBoost fusion method for recognising three dimensions of emotion, namely arousal (calm or excitement), valence (positive or negative feeling) and dominance (without control or empowered). We used a benchmark dataset called DREAMER where the EEG signals were collected from multiple stimulus along with self-evaluation ratings. In our proposed method, we first calculate the Short-Time Fourier Transform (STFT) of the EEG signals and convert them into RGB images to obtain the spectrograms. Then we use a two dimensional Convolutional Neural Network (CNN) in order to train the model on the spectrogram images and retrieve the features from the trained layer of the CNN using a dense layer of the neural network. We apply Extreme Gradient Boosting (XGBoost) classifier on extracted CNN features to classify the signals into arousal, valence and dominance of human emotion. We compare our results with the feature fusion-based state-of-the-art approaches of emotion recognition. To do this, we applied various feature extraction techniques on the signals which include Fast Fourier Transformation, Discrete Cosine Transformation, Poincare, Power Spectral Density, Hjorth parameters and some statistical features. Additionally, we use Chi-square and Recursive Feature Elimination techniques to select the discriminative features. We form the feature vectors by applying feature level fusion, and apply Support Vector Machine (SVM) and Extreme Gradient Boosting (XGBoost) classifiers on the fused features to classify different emotion levels. The performance study shows that the proposed spectrogram image based CNN-XGBoost fusion method outperforms the feature fusion-based SVM and XGBoost methods. The proposed method obtained the accuracy of 99.712% for arousal, 99.770% for valence and 99.770% for dominance in human emotion detection.

show abstract

Section: Related Workmentioning

confidence: 99%

CNN-XGBoost fusion-based affective state recognition using EEG spectrogram image analysis

Khan

Salsabil

Alam

et al. 2022

Sci Rep

View full text Add to dashboard Cite

show abstract

“…With EEG, ECG and EDA selected from AMIGOS, different features were extracted and then fed into different SVM classifiers to predict unimodal classification results. Dar et al [391] designed a 2D-CNN for EEG and combined LSTM and 1D-CNN for ECG and GSR. By using majority voting based on decisions made by multiple classifiers, the framework achieved the overall highest accuracy of 99.0% and 90.8% for AMIGOS [99] and DREAMER [321], respectively.…”

Section: Multi-physiological Modality Fusion For Affective Analysismentioning

confidence: 99%

A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances

Wang¹,

Song²,

Tao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Affective computing conjoins the research topics of emotion recognition and sentiment analysis, and can be realized with unimodal or multimodal data, consisting primarily of physical information (e.g., text, audio, and visual) and physiological signals (e.g., EEG and ECG). Physical-based affect recognition caters to more researchers due to the availability of multiple public databases, but it is challenging to reveal one's inner emotion hidden purposefully from facial expressions, audio tones, body gestures, etc. Physiological signals can generate more precise and reliable emotional results; yet, the difficulty in acquiring these signals hinders their practical application. Besides, by fusing physical information and physiological signals, useful features of emotional states can be obtained to enhance the performance of affective computing models. While existing reviews focus on one specific aspect of affective computing, we provide a systematical survey of important components: emotion models, databases, and recent advances. Firstly, we introduce two typical emotion models followed by five kinds of commonly used databases for affective computing. Next, we survey and taxonomize state-of-the-art unimodal affect recognition and multimodal affective analysis in terms of their detailed architectures and performances. Finally, we discuss some critical aspects of affective computing and its applications and conclude this review by pointing out some of the most promising future directions, such as the establishment of benchmark database and fusion strategies. The overarching goal of this systematic review is to help academic and industrial researchers understand the recent advances as well as new developments in this fast-paced, high-impact domain.

show abstract

“…Studies measuring stress by using various biological signals such as electroencephalography (EEG), electromyogram (EMG), oxygen saturation, and pulse waves have been published [3][4][5]. However, these measurement methods require expensive and bulky systems to acquire data, are complicated and expensive to use, and require signal analysis by experts.…”

Section: Introductionmentioning

confidence: 99%

Classification of Mental Stress Using CNN-LSTM Algorithms with Electrocardiogram Signals

Kang

Shin

Jung

et al. 2021

Journal of Healthcare Engineering

View full text Add to dashboard Cite

The mental stress faced by many people in modern society is a factor that causes various chronic diseases, such as depression, cancer, and cardiovascular disease, according to stress accumulation. Therefore, it is very important to regularly manage and monitor a person’s stress. In this study, we propose an ensemble algorithm that can accurately determine mental stress states using a modified convolutional neural network (CNN)- long short-term memory (LSTM) architecture. When a person is exposed to stress, a displacement occurs in the electrocardiogram (ECG) signal. It is possible to classify stress signals by analyzing ECG signals and extracting specific parameters. To maximize the performance of the proposed stress classification algorithm, fast Fourier transform (FFT) and spectrograms were applied to preprocess ECG signals and produce signals in both the time and frequency domains to aid the training process. As the performance evaluation benchmarks of the stress classification model, confusion matrices, receiver operating characteristic (ROC) curves, and precision-recall (PR) curves were used, and the accuracy achieved by the proposed model was 98.3%, which is an improvement of 14.7% compared to previous research results. Therefore, our model can help manage the mental health of people exposed to stress. In addition, if combined with various biosignals such as electromyogram (EMG) and photoplethysmography (PPG), it may have the potential for development in various healthcare systems, such as home training, sleep state analysis, and cardiovascular monitoring.

show abstract

CNN and LSTM-Based Emotion Charting Using Physiological Signals

Cited by 85 publications

References 61 publications

CNN-XGBoost fusion-based affective state recognition using EEG spectrogram image analysis

CNN-XGBoost fusion-based affective state recognition using EEG spectrogram image analysis

A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances

Classification of Mental Stress Using CNN-LSTM Algorithms with Electrocardiogram Signals

Contact Info

Product

Resources

About