Physiological Sensors Based Emotion Recognition While Experiencing Tactile Enhanced Multimedia

Abstract

Emotion recognition has increased the potential of affective computing by getting an instant feedback from users and thereby, have a better understanding of their behavior. Physiological sensors have been used to recognize human emotions in response to audio and video content that engages single (auditory) and multiple (two: auditory and vision) human senses, respectively. In this study, human emotions were recognized using physiological signals observed in response to tactile enhanced multimedia content that engages three (tactile, vision, and auditory) human senses. The aim was to give users an enhanced real-world sensation while engaging with multimedia content. To this end, four videos were selected and synchronized with an electric fan and a heater, based on timestamps within the scenes, to generate tactile enhanced content with cold and hot air effect respectively. Physiological signals, i.e., electroencephalography (EEG), photoplethysmography (PPG), and galvanic skin response (GSR) were recorded using commercially available sensors, while experiencing these tactile enhanced videos. The precision of the acquired physiological signals (including EEG, PPG, and GSR) is enhanced using pre-processing with a Savitzky-Golay smoothing filter. Frequency domain features (rational asymmetry, differential asymmetry, and correlation) from EEG, time domain features (variance, entropy, kurtosis, and skewness) from GSR, heart rate and heart rate variability from PPG data are extracted. The K nearest neighbor classifier is applied to the extracted features to classify four (happy, relaxed, angry, and sad) emotions. Our experimental results show that among individual modalities, PPG-based features gives the highest accuracy of 78.57 % as compared to EEG- and GSR-based features. The fusion of EEG, GSR, and PPG features further improved the classification accuracy to 79.76 % (for four emotions) when interacting with tactile enhanced multimedia.

DRER: Deep Learning–Based Driver’s Real Emotion Recognizer

“…There are many previous studies in the field of emotion recognition using bio-physiological signals, which involved the above sensors. Raheel et al [ 18 ] tried to recognize emotion while watching tactile enhanced multimedia. Four different video clips were selected and subjects had to rate the clips on a nine-point SAM scale about the videos.…”

“…Ultimately, the driver’s real emotion that we aim for is not concealed emotion but emotion that is not fully revealed. Furthermore, most research uses bio-physiology signals for recognizing human emotions [ 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ]. The most commonly used bio-physiology signals are electroencephalogram (EEG), electrocardiogram (ECG), photoplethysmography (PPG) and electrodermal activity (EDA).…”

Correlation Analysis of Different Measurement Places of Galvanic Skin Response in Test Groups Facing Pleasant and Unpleasant Stimuli

“…In this way, some literature, such as [ 8 ], has proposed three essential sources of emotion detection for a smart environment: facial emotion, behavior detection, and valence/arousal detection. Nowadays, wearable devices may help valence/arousal detection, incorporating measure features such as electrocardiography (ECG), electromyography (EMG), electroencephalography (EEG), galvanic skin response (GSR), photoplethysmography (PPG), and skin temperature (ST), as per many literature reports [ 9 , 10 , 11 , 12 , 13 ]. Although multimodal measures to improve emotion recognition performance are highly recommended [ 11 ], this study focuses on evaluating the performance of GSR signals gathered from wearable sensors in different measurement places.…”

“…Nowadays, wearable devices may help valence/arousal detection, incorporating measure features such as electrocardiography (ECG), electromyography (EMG), electroencephalography (EEG), galvanic skin response (GSR), photoplethysmography (PPG), and skin temperature (ST), as per many literature reports [ 9 , 10 , 11 , 12 , 13 ]. Although multimodal measures to improve emotion recognition performance are highly recommended [ 11 ], this study focuses on evaluating the performance of GSR signals gathered from wearable sensors in different measurement places. Recent advances in machine learning techniques in the last decade and the miniaturization of hardware technologies have inspired researchers to keep working on new ways to improve human activity recognition through emotion recognition using GSR sensor technology.…”

Correlation Analysis of Different Measurement Places of Galvanic Skin Response in Test Groups Facing Valence and Arousal Changes

“…This technology may incorporate features as Electrocardiography (ECG), Electromyography (EMG), Electroencephalography (EEG), Galvanic Skin Response (GSR) Photoplethysmography (PPG), and Skin Temperature (ST) as many literature reports [9][10] [11] [12] [13]. Although multimodal measures to improve emotion recognition performance are highly recommended [11], this study focuses on evaluating performance over GSR signals gathered from some wearables technology to improve autistic people's life quality. So simplicity and portability is part of the requirement for scenarios like these, where solutions should be as more transparent in individual daily life as possible.…”