Mode Variational LSTM Robust to Unseen Modes of Variation: Application to Facial Expression Recognition

Abstract

Spatio-temporal feature encoding is essential for encoding the dynamics in video sequences. Recurrent neural networks, particularly long short-term memory (LSTM) units, have been popular as an efficient tool for encoding spatio-temporal features in sequences. In this work, we investigate the effect of mode variations on the encoded spatio-temporal features using LSTMs. We show that the LSTM retains information related to the mode variation in the sequence, which is irrelevant to the task at hand (e.g. classification facial expressions). Actually, the LSTM forget mechanism is not robust enough to mode variations and preserves information that could negatively affect the encoded spatio-temporal features. We propose the mode variational LSTM to encode spatio-temporal features robust to unseen modes of variation. The mode variational LSTM modifies the original LSTM structure by adding an additional cell state that focuses on encoding the mode variation in the input sequence. To efficiently regulate what features should be stored in the additional cell state, additional gating functionality is also introduced. The effectiveness of the proposed mode variational LSTM is verified using the facial expression recognition task. Comparative experiments on publicly available datasets verified that the proposed mode variational LSTM outperforms existing methods. Moreover, a new dynamic facial expression dataset with different modes of variation, including various modes like pose and illumination variations, was collected to comprehensively evaluate the proposed mode variational LSTM. Experimental results verified that the proposed mode variational LSTM encodes spatio-temporal features robust to unseen modes of variation.

Hybrid Attention Cascade Network for Facial Expression Recognition

“…The average classification accuracy obtained in the experiments on the CK+, AFEW, and Oulu-CASIA datasets is shown in Tables 2-4, respectively. Table 2 shows the accuracy of our proposed method and state-of-the-art methods [14,18,24,31,33] on the AFEW dataset. On the AFEW dataset, the data come from the natural environment, which is restricted by head deflection, illumination, and blur.…”

“…Mode variational LSTM [31] 48.83% MRAN [14] 49.01% CNN-BLSTM [33] 49.09% FAN [18] 51.18% DenseNet-161 [24] 51.40%…”

“…By cascading CNN and RNN networks, facial expressions in video can be classified. Here, features are extracted by the CNN, and the extracted features are input into RNN training to estimate the expression classification of the entire video [ 31 , 32 ]. Cai et al used a cascaded network of a CNN and bidirectional long short-term memory (LSTM), and they completed the FER task in a natural environment through dual channel fusion (audio and visual) [ 33 ].…”

Improving Financial Time Series Prediction Accuracy Using Ensemble Empirical Mode Decomposition and Recurrent Neural Networks

“…Generally in a proportion (p, 1 − p) : p ∈ [0, 1] of 80 − 20%, such as is displayed in table 2. It corresponds to a common approach found in the recurrent neural network literature [2], [3], [36], which provides a set of independent and identical distributed training samples. A property that it is suitable in the analysis of individual model results.…”

“…forcing the model to retain irrelevant information [2]. In this regard, a myriad number of hybrid models in recent literature combining LSTM with other procedures manages to reduce the impact of the noise on the prediction of their model accuracy.…”