CorrFeat: Correlation-based Feature Extraction Algorithm using Skin Conductance and Pupil Diameter for Emotion Recognition

Zhang, Tianyi; Ali, Abdallah El; Wang, Chen; Zhu, Xintong; César, Pablo

doi:10.1145/3340555.3353716

Cited by 11 publications

(8 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this stage, intra-modality features

and

are fused using a correlation-based feature extraction method [ 82 ]. The purpose of correlation-based feature extraction is to extract features which (a) maximize the correlation coefficient between two modalities and (b) fuse the features between different instances.…”

Section: Methodsmentioning

confidence: 99%

CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors

Zhang

Ali

Wang³

et al. 2020

Sensors

Self Cite

View full text Add to dashboard Cite

Recognizing user emotions while they watch short-form videos anytime and anywhere is essential for facilitating video content customization and personalization. However, most works either classify a single emotion per video stimuli, or are restricted to static, desktop environments. To address this, we propose a correlation-based emotion recognition algorithm (CorrNet) to recognize the valence and arousal (V-A) of each instance (fine-grained segment of signals) using only wearable, physiological signals (e.g., electrodermal activity, heart rate). CorrNet takes advantage of features both inside each instance (intra-modality features) and between different instances for the same video stimuli (correlation-based features). We first test our approach on an indoor-desktop affect dataset (CASE), and thereafter on an outdoor-mobile affect dataset (MERCA) which we collected using a smart wristband and wearable eyetracker. Results show that for subject-independent binary classification (high-low), CorrNet yields promising recognition accuracies: 76.37% and 74.03% for V-A on CASE, and 70.29% and 68.15% for V-A on MERCA. Our findings show: (1) instance segment lengths between 1–4 s result in highest recognition accuracies (2) accuracies between laboratory-grade and wearable sensors are comparable, even under low sampling rates (≤64 Hz) (3) large amounts of neutral V-A labels, an artifact of continuous affect annotation, result in varied recognition performance.

show abstract

“…In this stage, intra-modality features

and

Section: Methodsmentioning

confidence: 99%

CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors

Zhang

Ali

Wang³

et al. 2020

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…Other physiological signals used include: electrocardiography (ECG) [ 19 – 21 ], electromyography (EMG) [ 22 ], electrodermal activity (EDA) [ 19 , 20 , 23 ], heart rate [ 24 – 26 ], respiration rate and depth [ 24 , 27 ], and arterial pressure [ 24 ]. Eye-tracking [ 28 – 32 ] and pupil width [ 33 – 36 ] are also used to recognize emotions.…”

Section: Introductionmentioning

confidence: 99%

Eye-Tracking Analysis for Emotion Recognition

Tarnowski

Kołodziej

Majkowski

et al. 2020

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

This article reports the results of the study related to emotion recognition by using eye-tracking. Emotions were evoked by presenting a dynamic movie material in the form of 21 video fragments. Eye-tracking signals recorded from 30 participants were used to calculate 18 features associated with eye movements (fixations and saccades) and pupil diameter. To ensure that the features were related to emotions, we investigated the influence of luminance and the dynamics of the presented movies. Three classes of emotions were considered: high arousal and low valence, low arousal and moderate valence, and high arousal and high valence. A maximum of 80% classification accuracy was obtained using the support vector machine (SVM) classifier and leave-one-subject-out validation method.

show abstract

“…Theoretically, the end-to-end model should result in the best performance as the features are directly connected with the ground truth labels [68], which means the deep representation is trained to best recognize these labels. However, according to previous studies [3], [69], if we train the network using fine-grained emotion labels and fully-supervised learning methods, the end-to-end model will overfit because of the temporal resolution mismatch between physiological signals and fine-grained self-reports due to different interoception levels across individuals [70]. Thus, we compare these three types of methods to find out whether the endto-end, deep feature extraction (deepfeat, section 3.2.1) still has the problem of overfitting for weakly-supervised learning.…”

Section: Feature Extraction Layersmentioning

confidence: 95%

“…The pairwise correlation-based features (pcorrfeat) are extracted by maximizing the correlation coefficient for every two signals from users who watch the same video stimuli [69]. The idea is inspired by the hypothesis that the same stimuli will trigger relatively similar emotions across physiological responses among different users [77], [78].…”

Section: Pcorrfeatmentioning

confidence: 99%

Weakly-Supervised Learning for Fine-Grained Emotion Recognition Using Physiological Signals

Zhang

Ali

Wang³

et al. 2023

IEEE Trans. Affective Comput.

Self Cite

View full text Add to dashboard Cite

Instead of predicting just one emotion for one activity (e.g., video watching), fine-grained emotion recognition enables more temporally precise recognition. Previous works on fine-grained emotion recognition require segment-by-segment, fine-grained emotion labels to train the recognition algorithm. However, experiments to collect these labels are costly and time-consuming compared with only collecting one emotion label after the user watched that stimulus (i.e., the post-stimuli emotion labels). To recognize emotions at a finer granularity level when trained with only post-stimuli labels, we propose an emotion recognition algorithm based on Deep Multiple Instance Learning (EDMIL ) using physiological signals. EDMIL recognizes fine-grained valence and arousal (V-A) labels by identifying which instances represent the post-stimuli V-A annotated by users after watching the videos. Instead of fully-supervised training, the instances are weakly-supervised by the post-stimuli labels in the training stage. The V-A of instances are estimated by the instance gains, which indicate the probability of instances to predict the post-stimuli labels. We tested EDMIL on three different datasets, CASE, MERCA and CEAP-360VR, collected in three different environments: desktop, mobile and HMD-based Virtual Reality, respectively. Recognition results validated with the fine-grained V-A self-reports show that for subject-independent 3-class classification (high/neutral/low), EDMIL obtains promising recognition accuracies: 75.63% and 79.73% for V-A on CASE, 70.51% and 67.62% for V-A on MERCA and 65.04% and 67.05% for V-A on CEAP-360VR. Our ablation study shows that all components of EDMIL contribute to both the classification and regression tasks. Our experiments also show that (1) compared with fullysupervised learning, weakly-supervised learning can reduce the problem of overfitting caused by the temporal mismatch between fine-grained annotations and physiological signals, (2) instance segment lengths between 1-2s result in the highest recognition accuracies and (3) EDMIL performs best if post-stimuli annotations consist of less than 30% or more than 60% of the entire video watching.

show abstract

CorrFeat: Correlation-based Feature Extraction Algorithm using Skin Conductance and Pupil Diameter for Emotion Recognition

Cited by 11 publications

References 31 publications

CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors

CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors

Eye-Tracking Analysis for Emotion Recognition

Weakly-Supervised Learning for Fine-Grained Emotion Recognition Using Physiological Signals

Contact Info

Product

Resources

About