Exploring Personal Memories and Video Content as Context for Facial Behavior in Predictions of Video-Induced Emotions

Abstract

Empirical evidence suggests that the emotional meaning of facial behavior in isolation is often ambiguous in real-world conditions.While humans complement interpretations of others' faces with additional reasoning about context, automated approaches rarely display such context-sensitivity. Empirical findings indicate that the personal memories triggered by videos are crucial for predicting viewers' emotional response to such videos -in some cases, even more so than the video's audiovisual content. In this article, we explore the benefits of personal memories as context for facial behavior analysis. We conduct a series of multimodal machine learning experiments combining the automatic analysis of video-viewers' faces with that of two types of context information for affective predictions: (1) self-reported free-text descriptions of triggered memories and (2) a video's audiovisual content. Our results demonstrate that both sources of context provide models with information about variation in viewers' affective responses that complement facial analysis and each other.CCS Concepts: • Human-centered computing → Empirical studies in ubiquitous and mobile computing.

Collecting Mementos: A Multimodal Dataset for Context-Sensitive Modeling of Affect and Memory Processing in Responses to Videos

“…Consequently, it may be challenging for automatic approaches that rely on such expressions to make accurate inferences. This conclusion is further supported by findings from our own prior experiments in which we extracted a broad range of affective lexical features from descriptions to predict induced emotions [9], [10] (see also Section 6 below). Confidence of Human Interpretations.…”

“…They demonstrate that the dataset successfully captures different aspects of affect and memory processing in response to videos and underlines how these relate to existing research in psychology. Some of the findings discussed in this section are presented in greater detail in other publications using the Mementos dataset [8], [9], [10]. 6…”

“…Additionally, it contains recordings of their behavior while doing so. We have used this corpus in previous research to model the contextual influence of occurring personal memories on the emotional impact of videos [8], [9], [10]. However, we believe that it can benefit future computational work on affect and memory processing more broadly, facilitating novel research beyond our initial inquiries.…”

The 4th Workshop on Modeling Socio-Emotional and Cognitive Processes from Multimodal Data In-the-Wild (MSECP-Wild)

“…A key challenge for robust predictions of internal states in realworld applications relates to the broad range of changing contextual conditions under which interactions with such systems occur [7]. However, despite the benefts of addressing context for predictions in the wild [4], existing technological research has only scarcely addressed this issue. Instead, eforts have largely focused on the context-free analysis of human behavioral signals.…”

End-to-End Continuous Speech Emotion Recognition in Real-life Customer Service Call Center Conversations

“…[7] provided an overview of context information (receivable elements and their knowledge and experience) 1 in audio-visual affective databases. [8] specifically discussed the influence of personal memories as a context for predicting emotional responses. [5] has shown the positive impact of contextual knowledge (action and relationship/environment context) on the affect-sensitive system.…”