Time-Delay Neural Network for Continuous Emotional Dimension Prediction From Facial Expression Sequences

Abstract

Automatic continuous affective state prediction from naturalistic facial expression is a very challenging research topic but very important in human-computer interaction. One of the main challenges is modeling the dynamics that characterize naturalistic expressions. In this paper, a novel two-stage automatic system is proposed to continuously predict affective dimension values from facial expression videos. In the first stage, traditional regression methods are used to classify each individual video frame, while in the second stage, a time-delay neural network (TDNN) is proposed to model the temporal relationships between consecutive predictions. The two-stage approach separates the emotional state dynamics modeling from an individual emotional state prediction step based on input features. In doing so, the temporal information used by the TDNN is not biased by the high variability between features of consecutive frames and allows the network to more easily exploit the slow changing dynamics between emotional states. The system was fully tested and evaluated on three different facial expression video datasets. Our experimental results demonstrate that the use of a two-stage approach combined with the TDNN to take into account previously classified frames significantly improves the overall performance of continuous emotional state estimation in naturalistic facial expressions. The proposed approach has won the affect recognition sub-challenge of the Third International Audio/Visual Emotion Recognition Challenge.

Unobtrusive Inference of Affective States in Virtual Rehabilitation from Upper Limb Motions: A Feasibility Study

“…A number of machine learning approaches have been used to continuously track information over time and recognize affective states [22], [23], [24], [25], [26], [27]. The temporal variation is an important issue in recognizing naturalistic everyday affective states [28].…”

“…SVM is a popular choice in many affective recognition systems, so it is used here as baseline [29]. Other classifiers that have been employed include neural networks [22], recurrent neural networks [24], [30], dynamic Bayesian networks [27], hidden Markov models [25], [26], [31], [32], latent-dynamic conditional random fields [33], [34]. The Naïve Bayes classifiers has been studied and compared with others classifiers and has often been more effective than sophisticated rules [35], [36], [37], [38].…”

Improvement of Anomalous Behavior Detection of GNSS Signal Based on TDNN for Augmentation Systems

“…An input signal propagates through the network in the direction of the connections until it reaches the output of the network. In supervised learning, the learning algorithm adjusts the weights to minimize the discrepancy between the output of the network and the desired value provided [ 20 , 21 , 22 ].…”

Linear and Non-Linear Multimodal Fusion for Continuous Affect Estimation In-the-Wild

“…Support Vector Machine (SVR) is perhaps the most widely used regression method for affect estimation and has been regarded as baseline approach for affect estimation [3] [15] [11]. Recent literature takes into account short term temporal correlation such as Continuous Conditional Random Fields (CCRF) on top of SVRs [16] and various type of neural network including Time Delay Neural Networks [17], Recurrent Neural Networks (RNN) [18] and Long-Short Term Memory RNN (LSTM-RNN) in [19] [20]. Another study [12], employed a bidirectional LSTM model with an outputassociative framework to achieve improved performance in affect prediction.…”