Facial expression recognition based on regional adaptive correlation

Abstract

To address the problem that the features extracted by CNN‐based facial expression recognition (FER) do not consider structural information, a region adaptive correlation deep network (RACN) is proposed. The network consists of two branches. In one branch, the features obtained by applying CNN to facial sub‐blocks are used as the input of the proposed second‐order region correlation network (SRCN), which obtains structural features by adaptively learning the correlation of facial regions. Furthermore, they are fused with the parallel branch‐extracted global features to obtain a comprehensive high‐semantic feature representation. Finally, weights are assigned to the two features through the channel attention mechanism for more accurate expression classification. Experimental results show that our method can effectively extract expression features in an end‐to‐end manner, improve the accuracy of FER, and achieve competitive performance without relying on any a priori knowledge. And the region‐adaptive correlation feature extraction branch RACN can be applied to other deep learning networks to extract discriminative structural‐adaptive features. To the best of our knowledge, our work is the first to enrich the feature representation for end‐to‐end static FER by adaptively obtaining more discriminative regional adaptive correlation feature vectors via the autocorrelation matrix combined with CNN compared to the existing literature.

Target Perception and Behavioral Recognition Algorithms Based on Saliency and Feature Extraction

“…(11) where the grey level of the original image is L. where is the salient class mean: (12) The background mean: (13) That is, the overall mean of the whole image is μ. (14) i.e. the intra-class variance can be obtained as (15) The inter-class variance is 𝜎 (𝑇).…”

Algorithms used for facial emotion recognition: a systematic review of the literature

“…If the output of an individual node is above the specified threshold value, that node becomes active and sends data to the next layer in the network. Otherwise, no data is passed to the next network layer (25). Convolutional neural networks are distinguished from other neural networks by their superior performance with image, speech, or audio signal inputs.…”

“…The first layers focus on simple features, such as colors and edges. As the image data progresses through the layers, the CNN begins to recognize larger elements or shapes until it finally identifies the expected object (25).…”

“…The feature detector is a twodimensional (2D) array of weights representing a portion of the image. Although its size can vary, the size of the filter is usually a 3x3 matrix; this also determines the size of the receptive field (25). A Convolutional Neural Network (CNN) has one or more convolutional layers, they are grouping layers and fully connected and are used in image recognition.…”

“…As mentioned above, the pixel values of the input image are not directly connected to the output layer in partially connected layers. However, in the fully connected layer, each node of the output layer is directly connected to a node of the previous layer (25).…”