Muse-ing on the Impact of Utterance Ordering on Crowdsourced Emotion Annotations

Abstract

Emotion recognition algorithms rely on data annotated with high quality labels. However, emotion expression and perception are inherently subjective. There is generally not a single annotation that can be unambiguously declared "correct." As a result, annotations are colored by the manner in which they were collected. In this paper, we conduct crowdsourcing experiments to investigate this impact on both the annotations themselves and on the performance of these algorithms. We focus on one critical question: the effect of context. We present a new emotion dataset, Multimodal Stressed Emotion (MuSE), and annotate the dataset using two conditions: randomized, in which annotators are presented with clips in random order, and contextualized, in which annotators are presented with clips in order. We find that contextual labeling schemes result in annotations that are more similar to a speaker's own self-reported labels and that labels generated from randomized schemes are most easily predictable by automated systems.

PAGAN: Video Affect Annotation Made Easy

“…Especially for frequent videogame players, who have personal experiences with the dynamics of shooter games, this video is easier to interpret and the affective high-points are easier to recognise. This is also supported by a recent study of Jaiswal et al [40], who also observed an effect between the context of the annotation task and the quality of labels.…”

Privacy Enhanced Multimodal Neural Representations for Emotion Recognition

“…We use four common emotion recognition datasets: MSP-Improv (Busso et al 2017), MSP-Podcast (Lotfian and Busso 2017), Interactive Emotional Dyadic MOtion Capture (IEMOCAP) dataset (Busso et al 2008), and Multimodal Stressed Emotion (MuSE) dataset (Jaiswal et al 2019).…”

Controlling for Confounders in Multimodal Emotion Classification via Adversarial Learning

“…We use three datasets to study the effect of stress on emotion recognition: (1) Multimodal Stressed Emotion (MuSE) dataset [19]; (2) Interactive Emotional Dyadic MOtion Capture (IEMOCAP) dataset [6]; and (3) MSP-Improv dataset [7].…”

“…Emotion Labels. Each utterance in the MuSE dataset was labeled for activation and valence on a nine-point Likert scale by eight crowd-sourced annotators [19], who observed the data in random order across subjects. We average the annotations to obtain a mean score for each utterance, and then bin the mean score into one of three classes, defined as, {"low": [min, 4.5], "mid": (4.5, 5.5], "high": (5.5, max]}.…”