CEAP-360VR: A Continuous Physiological and Behavioral Emotion Annotation Dataset for 360$^\circ$ VR Videos

Xue, Ting; Ali, Abdallah El; Zhang, Tianyi; Ding, Guofu; César, Pablo

doi:10.1109/tmm.2021.3124080

Cited by 27 publications

(21 citation statements)

References 85 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To evaluate the performance of EDMIL, we test it on three datasets: CASE [15], MERCA [9] and CEAP-360VR [82] collected in three environments: desktop, mobile and HMD-based Virtual Reality, respectively. EDMIL is an end-to-end weaklysupervised learning algorithm for modeling the temporal ambiguity of emotions.…”

Section: Datasetsmentioning

confidence: 99%

Weakly-Supervised Learning for Fine-Grained Emotion Recognition Using Physiological Signals

Zhang

Ali

Wang³

et al. 2023

IEEE Trans. Affective Comput.

Self Cite

View full text Add to dashboard Cite

Instead of predicting just one emotion for one activity (e.g., video watching), fine-grained emotion recognition enables more temporally precise recognition. Previous works on fine-grained emotion recognition require segment-by-segment, fine-grained emotion labels to train the recognition algorithm. However, experiments to collect these labels are costly and time-consuming compared with only collecting one emotion label after the user watched that stimulus (i.e., the post-stimuli emotion labels). To recognize emotions at a finer granularity level when trained with only post-stimuli labels, we propose an emotion recognition algorithm based on Deep Multiple Instance Learning (EDMIL ) using physiological signals. EDMIL recognizes fine-grained valence and arousal (V-A) labels by identifying which instances represent the post-stimuli V-A annotated by users after watching the videos. Instead of fully-supervised training, the instances are weakly-supervised by the post-stimuli labels in the training stage. The V-A of instances are estimated by the instance gains, which indicate the probability of instances to predict the post-stimuli labels. We tested EDMIL on three different datasets, CASE, MERCA and CEAP-360VR, collected in three different environments: desktop, mobile and HMD-based Virtual Reality, respectively. Recognition results validated with the fine-grained V-A self-reports show that for subject-independent 3-class classification (high/neutral/low), EDMIL obtains promising recognition accuracies: 75.63% and 79.73% for V-A on CASE, 70.51% and 67.62% for V-A on MERCA and 65.04% and 67.05% for V-A on CEAP-360VR. Our ablation study shows that all components of EDMIL contribute to both the classification and regression tasks. Our experiments also show that (1) compared with fullysupervised learning, weakly-supervised learning can reduce the problem of overfitting caused by the temporal mismatch between fine-grained annotations and physiological signals, (2) instance segment lengths between 1-2s result in the highest recognition accuracies and (3) EDMIL performs best if post-stimuli annotations consist of less than 30% or more than 60% of the entire video watching.

show abstract

Section: Datasetsmentioning

confidence: 99%

Weakly-Supervised Learning for Fine-Grained Emotion Recognition Using Physiological Signals

Zhang

Ali

Wang³

et al. 2023

IEEE Trans. Affective Comput.

Self Cite

View full text Add to dashboard Cite

show abstract

“…In [ 49 ], participants viewed a 360° videos as stimuli and they rated emotional states (valence and arousal) continuously using the joystick. It is observed that collecting continuous annotations can be used to evaluate the performance of fine-grained emotion recognition algorithms such as weakly supervised learning or regression.…”

Section: Data Annotationmentioning

confidence: 99%

A Survey on Physiological Signal-Based Emotion Recognition

Ahmad

Khan

2022

Bioengineering

View full text Add to dashboard Cite

Physiological signals are the most reliable form of signals for emotion recognition, as they cannot be controlled deliberately by the subject. Existing review papers on emotion recognition based on physiological signals surveyed only the regular steps involved in the workflow of emotion recognition such as pre-processing, feature extraction, and classification. While these are important steps, such steps are required for any signal processing application. Emotion recognition poses its own set of challenges that are very important to address for a robust system. Thus, to bridge the gap in the existing literature, in this paper, we review the effect of inter-subject data variance on emotion recognition, important data annotation techniques for emotion recognition and their comparison, data pre-processing techniques for each physiological signal, data splitting techniques for improving the generalization of emotion recognition models and different multimodal fusion techniques and their comparison. Finally, we discuss key challenges and future directions in this field.

show abstract

“…In [39], participant viewed a 360 • videos as stimuli and they rated emotional states (valence and arousal) continuously using the joystick. It is observed that collecting continuous annotations can be used to evaluate the performance of finegrained emotion recognition algorithms such as weakly supervised learning or regression.…”

Section: B Continuous Annotationmentioning

confidence: 99%

“…These measurements can be observational or self-reported and are provided in real-time Fig. 5: Continuous annotation using HCI mechanism [39] through the manipulation of a computer joystick.…”

Section: B Continuous Annotationmentioning

confidence: 99%

A Survey on Physiological Signal Based Emotion Recognition

Ahmad¹,

Khan²

2022

Preprint

View full text Add to dashboard Cite

Physiological Signals are the most reliable form of signals for emotion recognition, as they cannot be controlled deliberately by the subject. Existing review papers on emotion recognition based on physiological signals surveyed only the regular steps involved in the workflow of emotion recognition such as preprocessing, feature extraction, and classification. While these are important steps, such steps are required for any signal processing application. Emotion recognition poses its own set of challenges that are very important to address for a robust system. Thus, to bridge the gap in the existing literature, in this paper, we review the effect of inter-subject data variance on emotion recognition, important data annotation techniques for emotion recognition and their comparison, data preprocessing techniques for each physiological signal, data splitting techniques for improving the generalization of emotion recognition models and different multimodal fusion techniques and their comparison. Finally we discuss key challenges and future directions in this field.

show abstract

CEAP-360VR: A Continuous Physiological and Behavioral Emotion Annotation Dataset for 360$^\circ$ VR Videos

Cited by 27 publications

References 85 publications

Weakly-Supervised Learning for Fine-Grained Emotion Recognition Using Physiological Signals

Weakly-Supervised Learning for Fine-Grained Emotion Recognition Using Physiological Signals

A Survey on Physiological Signal-Based Emotion Recognition

A Survey on Physiological Signal Based Emotion Recognition

Contact Info

Product

Resources

About