Modeling Multiple Temporal Scales of Full-Body Movements for Emotion Classification

Abstract

This work investigates classification of emotions from full-body movements by using a novel Convolutional Neural Network-based architecture. The model is composed of two shallow networks processing in parallel when the 8-bit RGB images obtained from time intervals of 3D-positional data are the inputs. One network performs a coarse-grained modelling in the time domain while the other one applies a fine-grained modelling. We show that combining different temporal scales into a single architecture improves the classification results of a dataset composed of short excerpts of the performances of professional dancers who interpreted four affective states: anger, happiness, sadness, and insecurity. Additionally, we investigate the effect of data chunk duration, overlapping, the size of the input images and the contribution of several data augmentation strategies for our proposed method. Better recognition results were obtained when the duration of a data chunk was longer, and this was further improved by applying balanced data augmentation. Moreover, we test our method on other existing motion capture datasets and compare the results with prior art. In all experiments, our results surpassed the state-of-the-art approaches, showing that this method generalizes across diverse settings and contexts.

Graph Laplacian-Improved Convolutional Residual Autoencoder for Unsupervised Human Action and Emotion Recognition

“…a forward head and chest bend express sadness in [40]), ii) recognizing specific gestures which are emblems of the emotions (e.g. raising arms and hands-on-hips are the gestures of pride according to [41] and [42]), or iii) processing the expressive quality of the movement [6], [9], [43], [44]. Out of these three possibilities, the second and the third use the temporal information of the data, while the first one performs only spatial processing.…”

“…For example, [49] used an RNN with 3-layers to perform emotion classification from MoCap data of daily activities: clapping, drinking, throwing, and waving, etc., associated with four emotions: happy, angry, sad, and neutral. Beyan et al [6] present the joint training of two CNNs such that one of them performs coarse-grained modeling while the other applies fine-grained modeling in the time. The inputs of this network are 8-bit RGB images obtained from 3D-skeleton data over time.…”

“…On the other hand, the analysis of human emotions can include several modalities, such as text, physiological signals, acoustic data, facial landmarks, facial images, or full-body motion. Among all, recent HER works [6], [7], [8], [9], [10] highlight the effectiveness of processing full-body motion represented in terms of 3D-skeleton data.…”

“…The successes in skeleton-based HAR [11], [12], [13] and skeleton-based HER [6], [7], [8], [9], [10] primarily leverage on the supervised learning paradigm, which requires a significant amount of manually labeled data. However, data annotation is notoriously expensive and time-consuming [14].…”

Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss

“…The majority of works mainly concentrated on unimodal learning of emotions [11], [12], [13], i.e., processing a single modality. Although there exist breakthrough achievements by unimodal emotion recognition, due to the aforementioned multimodal nature of emotion expression, such models remain incapable in some circumstances.…”