MIMAMO Net: Integrating Micro- and Macro-Motion for Video Emotion Recognition

Abstract

Spatial-temporal feature learning is of vital importance for video emotion recognition. Previous deep network structures often focused on macro-motion which extends over long time scales, e.g., on the order of seconds. We believe integrating structures capturing information about both micro- and macro-motion will benefit emotion prediction, because human perceive both micro- and macro-expressions. In this paper, we propose to combine micro- and macro-motion features to improve video emotion recognition with a two-stream recurrent network, named MIMAMO (Micro-Macro-Motion) Net. Specifically, smaller and shorter micro-motions are analyzed by a two-stream network, while larger and more sustained macro-motions can be well captured by a subsequent recurrent network. Assigning specific interpretations to the roles of different parts of the network enables us to make choice of parameters based on prior knowledge: choices that turn out to be optimal. One of the important innovations in our model is the use of interframe phase differences rather than optical flow as input to the temporal stream. Compared with the optical flow, phase differences require less computation and are more robust to illumination changes. Our proposed network achieves state of the art performance on two video emotion datasets, the OMG emotion dataset and the Aff-Wild dataset. The most significant gains are for arousal prediction, for which motion information is intuitively more informative. Source code is available at https://github.com/wtomin/MIMAMO-Net.

Functional neural network for decision processing, a racing network of programmable neurons with fuzzy logic where the target operating model relies on the network itself

“…Each of them is a programmable single unit which has multiple inputs. The inputs of each neuron are obtained at the time of launching a new request by using a simple cognitive test that can quantify the subject's reaction time, combined with the MIMAMO-net (Deng et al 2019) that can qualify the subject's emotional response by facial emotion recognition. In other words, FNN is an AI model based on a network of specialized neurons, i.e.…”

Deep Emotion Recognition using Facial, Speech and Textual Cues: A Survey

“…Specifically, Pre-trained 3D-CNN for Human Action is adopted to extract spatiotemporal features of emotions in videos [52][53]. Furthermore, Pre-trained 3D-CNN for sports are also applied in a series of studies [7] [58], which consists of two stages: Two-Stream convolutional Neural Network and Gated Recurrent Unit Network to capture both micro-and macro-motion, respectively. The feature representation of a snippet, i.e.…”

“…• W eighted average F 1: (1,1) (0,0) (0.5,0.5) (0.56,0. 45) [130] (0.529, 0.377) [58] (0.535, 0.365) [41] (Valence, Arousal) 2019 2020 2021 (d) OMG Fig. 6.…”

Multimodal Emotion Recognition Using Non-Inertial Loss Function

“…There are many related works in this field depending on data sets used and models proposed [20], [21]. There are also previous studies on emotion recognition in videos [18], [25]. Valence and Arousal (V-A) are not separated values; binding these two parameters describe an emotion.…”

“…Vielzeuf et al [14] trained audiovisual ensemble network on emotion video classification. MI-MAMO Net Deng et al [18] trained a spatial, temporal network with CCC loss.…”