RCEA: Real-time, Continuous Emotion Annotation for Collecting Precise Mobile Video Ground Truth Labels

Zhang, Tianyi; Ali, Abdallah El; Wang, Chen; Hanjalic, Alan; César, Pablo

doi:10.1145/3313831.3376808

Cited by 35 publications

(46 citation statements)

References 92 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As shown in Figure 10 , more than

of samples from CASE and

of samples from MERCA belong to the neutral class. The resulting high amounts of neutral V-A ratings cannot be attributed to the mobile aspect of MERCA’s data collection, given that users spent most of their time (up to 73.2%) standing while watching and annotating [ 32 ]. We instead attribute this phenomenon to the act of annotating continuously, irrespective of environment (static vs. mobile).…”

Section: Discussionmentioning

confidence: 99%

“…To verify the validity of CorrNet using wearable physiological sensors, we collected continuous self-annotated physiological signals. Here, users annotated their valence and arousal levels using a continuous mobile annotation technique (cf., [ 32 ]) in a controlled, outdoor environment. This data collection resulted in the Mobile Emotion Recognition with Continuous Annotation (MERCA) dataset, which we describe below in Section 4.2 .…”

Section: Datasetsmentioning

confidence: 99%

“…Emotions (as V-A) are annotated by participants using a real-time, continuous emotion annotation (RCEA) mobile application [ 32 ]. Participants can input their valence and arousal using a virtual joystick (shown in Figure 5 ) on the screen of the mobile device which they use for video watching.…”

Section: Datasetsmentioning

confidence: 99%

“…Given the foregoing, we focus on fine-grained emotion recognition using wearable physiological sensors. To this end, we collected the Mobile Emotion Recognition with Continuous Annotation (MERCA) dataset, where users annotate their valence and arousal states using a continuous mobile annotation input technique (cf., [ 32 ]) in real-time while watching short-form videos.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors

Zhang

Ali

Wang³

et al. 2020

Sensors

Self Cite

View full text Add to dashboard Cite

Recognizing user emotions while they watch short-form videos anytime and anywhere is essential for facilitating video content customization and personalization. However, most works either classify a single emotion per video stimuli, or are restricted to static, desktop environments. To address this, we propose a correlation-based emotion recognition algorithm (CorrNet) to recognize the valence and arousal (V-A) of each instance (fine-grained segment of signals) using only wearable, physiological signals (e.g., electrodermal activity, heart rate). CorrNet takes advantage of features both inside each instance (intra-modality features) and between different instances for the same video stimuli (correlation-based features). We first test our approach on an indoor-desktop affect dataset (CASE), and thereafter on an outdoor-mobile affect dataset (MERCA) which we collected using a smart wristband and wearable eyetracker. Results show that for subject-independent binary classification (high-low), CorrNet yields promising recognition accuracies: 76.37% and 74.03% for V-A on CASE, and 70.29% and 68.15% for V-A on MERCA. Our findings show: (1) instance segment lengths between 1–4 s result in highest recognition accuracies (2) accuracies between laboratory-grade and wearable sensors are comparable, even under low sampling rates (≤64 Hz) (3) large amounts of neutral V-A labels, an artifact of continuous affect annotation, result in varied recognition performance.

show abstract

“…As shown in Figure 10 , more than

of samples from CASE and

Section: Discussionmentioning

confidence: 99%

Section: Datasetsmentioning

confidence: 99%

Section: Datasetsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors

Zhang

Ali

Wang³

et al. 2020

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…For inputting annotations continuously, prior research use either joystick-based controllers (e.g., DARMA [10] or CASE [32]), or a physical radial controller if specifying a single, continuous dimension such as emotional intensity (e.g., RankTrace [20]). Recently, Zhang et al [38] proposed RCEA, which is suitable for mobile touchscreens and mobile video watching scenarios. Given that in our case users will be wearing an HMD, we need to enable easy controller-based input that can be used while users' visual attention is occupied by the 360 • video content.…”

Section: Annotating Emotions Continuouslymentioning

confidence: 99%

Designing Real-time, Continuous Emotion Annotation Techniques for 360° VR Videos

Xue

Ghosh

Ding

et al. 2020

Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems

Self Cite

View full text Add to dashboard Cite

With the increasing availability of head-mounted displays (HMDs) that show immersive 360 • VR content, it is important to understand to what extent these immersive experiences can evoke emotions. Typically to collect emotion ground truth labels, users rate videos through postexperience self-reports that are discrete in nature. However, post-stimuli self-reports are temporally imprecise, especially after watching 360 • videos. In this work, we design six continuous emotion annotation techniques for the Oculus Rift HMD aimed at minimizing workload and distraction. Based on a co-design session with six experts, we contribute HaloLight and DotSize, two continuous annotation methods deemed unobtrusive and easy to understand. We discuss the next challenges for evaluating the usability of these techniques, and reliability of continuous annotations.

show abstract

Video Quality Prediction: An Exploratory Study With Valence and Arousal Signals

Di Tecco,

Foglia,

Prete

2024

IEEE Access

View full text Add to dashboard Cite

With the explosion of online video consumption, assessing and anticipating how users will evaluate the content they watch has become increasingly important. Traditional methods based on explicit user feedback are often limited in their ability to do this, as they can be time-consuming and expensive to collect. This study explores techniques to predict users' ratings about a video's ability to evoke emotions through emotional signals. In particular, it is proposed a method of emotional analysis that uses valence and arousal data as key signals for predicting user ratings through systems that use machine-learning techniques. Hence, an experiment in the wild involved 112 participants who completed questionnaires to create a dataset of emotional data and video quality ratings to train different intelligent systems. The best system comprised a Medium Gaussian Support Vector Machine (SVM) classifier that detected users' ratings between Ineffective and Effective based on valence and arousal features as input, achieving an accuracy higher than 87%. The result demonstrated that it is possible to predict users' ratings on the ability of the movie to elicit emotion, using users' emotional states in terms of valence and arousal. The system has several advantages, such as eliminating the need for user reports, predicting user ratings in real-time more quickly and dynamically, and utilizing only the initial emotional state to predict users' ratings. In addition, it has potential applications in advertising, education, and entertainment fields. Advertisers could better understand how consumers perceive their products and create more effective advertising campaigns; educational institutions could develop more engaging and effective learning materials; entertainment providers could create more popular and successful content.

show abstract

RCEA: Real-time, Continuous Emotion Annotation for Collecting Precise Mobile Video Ground Truth Labels

Cited by 35 publications

References 92 publications

CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors

CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors

Designing Real-time, Continuous Emotion Annotation Techniques for 360° VR Videos

Video Quality Prediction: An Exploratory Study With Valence and Arousal Signals

Contact Info

Product

Resources

About