Multimodal Emotion Evaluation: A Physiological Model for Cost-Effective Emotion Classification

Abstract

Emotional responses are associated with distinct body alterations and are crucial to foster adaptive responses, well-being, and survival. Emotion identification may improve peoples’ emotion regulation strategies and interaction with multiple life contexts. Several studies have investigated emotion classification systems, but most of them are based on the analysis of only one, a few, or isolated physiological signals. Understanding how informative the individual signals are and how their combination works would allow to develop more cost-effective, informative, and objective systems for emotion detection, processing, and interpretation. In the present work, electrocardiogram, electromyogram, and electrodermal activity were processed in order to find a physiological model of emotions. Both a unimodal and a multimodal approach were used to analyze what signal, or combination of signals, may better describe an emotional response, using a sample of 55 healthy subjects. The method was divided in: (1) signal preprocessing; (2) feature extraction; (3) classification using random forest and neural networks. Results suggest that the electrocardiogram (ECG) signal is the most effective for emotion classification. Yet, the combination of all signals provides the best emotion identification performance, with all signals providing crucial information for the system. This physiological model of emotions has important research and clinical implications, by providing valuable information about the value and weight of physiological signals for emotional classification, which can critically drive effective evaluation, monitoring and intervention, regarding emotional processing and regulation, considering multiple contexts.

Neuroorganoleptics: Organoleptic Testing Based on Psychophysiological Sensing

“…In order to validate the potential of the psychophysiological data dimension of this neuroorganoleptics protocol, the ECG signal was processed and analysed. We focus on this modality, given that heart rate variability (HRV) analysis is frequently present in emotion evaluation studies using psychophysiological data [ 16 , 34 , 35 , 36 ].…”

The Concept of Advanced Multi-Sensor Monitoring of Human Stress

“…A large number of scientists have been interested in the idea of using the coherence of several sensors to determine stress in their laboratories [ 148 , 180 , 181 , 182 ]. For example it is worth to mention the classifier of negative emotion induced by a visual stimulation evaluated from EDA, ECG and skin temperature [ 183 ], multimodal emotion evaluation from combination of EDA, ECG and EMG [ 184 ], driver anxiety detection using EDA, PPG, EEG and pupil information [ 185 ], identification of cognitive tasks by machine learning from EDA and HRV [ 186 ], evaluating of mental workload during web browsing from EDA, PPG and EEG [ 156 , 187 ]. Nowadays, the trend is the use of virtual reality [ 145 , 188 ].…”

TSN: Performance Creative Choreography Based on Twin Sensor Network

“…For example, it is difficult to achieve accurate statistics and quantification of specific behaviours in the game, which has a greater impact on the collection and evaluation of key information. Pinto et al [28] processed electrocardiogram, electromyography, and dermal electrical activity to find a physiological model of emotion. Using samples of 55 healthy subjects, Pinto et al used single-peak and multipeak methods to analyse which signals or combinations of signals can better describe emotional responses.…”