Decoding Emotion Dynamics in Videos using Dynamic Exploratory Graph Analysis and Zero-Shot Image Classification: A Simulation and Tutorial using the transforEmotion R package

Abstract

In this paper, we propose a novel approach for modeling and understanding the dynamics of emotion facial expression recognition (FER) scores. Recent advancements in deep learning and transformer-based neural network architectures enable the time series analysis of FER scores extracted from images and videos. This type of data can be important for psychological research of affective dynamics and emotion expression dynamics. However, the properties of such data are not well understood in the current literature. We propose a new method to simulate FER scores based on a modified version of the Damped Linear Oscillator with a measurement model (DLO-MM). We use this model to conduct a large-scale simulation and use dynamic Exploratory Graph Analysis to investigate the dimensionality of the data and use network scores to recover the values of the latent dimensions—positive and negative sentiment of the expressed emotions. Our results show that the DLO-MM model can be used to simulate FER scores for different patterns of emotion dynamics and that DynEGA can be used to uncover the latent structure of emotion dynamics expressed through FER scores. All methods presented in the paper are implemented in the transforEmotion R package and the tutorial section provides a step-by-step guide on how to simulate FER scores using DLO-MM and how to estimate FER scores from YouTube videos using transformer-based machine learning models.

When ELIZA Meets Therapists: A Turing Test for the Heart and Mind

“…transforEmotion was used to compute the probability that the responses written by human therapists and ChatGPT were more in line with the common factors of therapy. 31 Overall, responses written by ChatGPT were more likely to be classified as connecting (d = 1.00, 95% CI [0.40, 1.59]), empathic (d = 0.75, 95% CI [0.11, 1.40]), and culturally competent (0.81, 95% CI [0.17, 1.46]) than responses written by therapists. Although the effect for effectiveness was medium-sized, it was not reliably different from zero (d = 0.64, 95% CI [-0.03, 1.30]).…”