Improving the Accuracy of Automatic Facial Expression Recognition in Speaking Subjects with Deep Learning

Abstract

When automatic facial expression recognition is applied to video sequences of speaking subjects, the recognition accuracy has been noted to be lower than with video sequences of still subjects. This effect known as the speaking effect arises during spontaneous conversations, and along with the affective expressions the speech articulation process influences facial configurations. In this work we question whether, aside from facial features, other cues relating to the articulation process would increase emotion recognition accuracy when added in input to a deep neural network model. We develop two neural networks that classify facial expressions in speaking subjects from the RAVDESS dataset, a spatio-temporal CNN and a GRU cell RNN. They are first trained on facial features only, and afterwards both on facial features and articulation related cues extracted from a model trained for lip reading, while varying the number of consecutive frames provided in input as well. We show that using DNNs the addition of features related to articulation increases classification accuracy up to 12%, the increase being greater with more consecutive frames provided in input to the model.

LSTM and GRU Neural Networks as Models of Dynamical Processes Used in Predictive Control: A Comparison of Models Developed for Two Chemical Reactors

“…Example applications of the LSTM architecture are: data classification [ 22 ], speech recognition [ 23 , 24 ], handwriting recognition [ 25 ], speech synthesis [ 26 ], text coherence tests [ 27 ], biometric authentication and anomaly detection [ 28 ], detecting deception from gaze and speech [ 29 ] and anomaly detection [ 30 ]. Similarly, example applications of the GRU structure are: facial expression recognition [ 31 ], human activity recognition [ 32 ], cyberbullying detection [ 33 ], defect detection [ 34 ], human activity surveillance [ 35 ], automated classification of cognitive workload tasks [ 36 ] and speaker identification [ 37 ].…”

A CNN-LSTM Based Deep Neural Networks for Facial Emotion Detection in Videos

“…In [3]the research work has been carried on to determine the contribution of audiovisual cues in emotion recognition and how temporal relation between audio and video impacts the rate and speed on emotion recognition. In [6] work has been carried out on RAVDEES dataset [4]; the frames have been increased in the step of 5 starting from 5 frames/video till 65 frames/video. Two Neural Networks have been designed namely Spatial Temporal CNN [6]…”

“…In [6] work has been carried out on RAVDEES dataset [4]; the frames have been increased in the step of 5 starting from 5 frames/video till 65 frames/video. Two Neural Networks have been designed namely Spatial Temporal CNN [6]…”

“…In their fully connected layers, CNN [4] uses a mathematical operation called convolution that is used instead of normal matrix multiplication to generate the features from the input data. The CNN [5] which convolves image data through many filters and finally produces a feature map, further the feature map produced is combined with the fully connected layers of neurons and finally the Emotions are recognized as belonging to a particular class based on output [6]. Apart from this CNN can be used for different applications involving images/videos such as medical imagebased analysis, agricultural field (plant disease detection), Object Detection, Facial Recognition, Face detection etc.…”

Modelling and Analysis of Facial Expressions Using Optical Flow Derived Divergence and Curl Templates

“…In a video with subjects involved in a spontaneous conversation, the speech articulation process conspicuously influences facial configuration and has been observed to reduce the FER accuracy as compared to the case where the subjects are not talking. Bursic et al [27] noted that while examining FEs of subjects involved in such conversations, the speaking effect needs to be regarded as a crucial factor. They developed a deep neural network-based model that analyzed cues related to facial features and speech articulation extracted from a model trained for lipreading.…”