Adaptive Multilayer Perceptual Attention Network for Facial Expression Recognition

Abstract

In complex real-world situations, problems such as illumination changes, facial occlusion, and variant poses make facial expression recognition (FER) a challenging task. To solve the robustness problem, this paper proposes an adaptive multilayer perceptual attention network (AMP-Net) that is inspired by the facial attributes and the facial perception mechanism of the human visual system. AMP-Net extracts global, local, and salient facial emotional features with different fine-grained features to learn the underlying diversity and key information of facial emotions. Different from existing methods, AMP-Net can adaptively guide the network to focus on multiple finer and distinguishable local patches with robustness to occlusion and variant poses, improving the effectiveness of learning potential facial diversity information. In addition, the proposed global perception module can learn different receptive field features in the global perception domain, and AMP-Net also supplements salient facial region features with high emotion correlation based on prior knowledge to capture key texture details and avoid important information loss. Many experiments show that AMP-Net achieves good generalizability and state-of-the-art results on several real-world datasets, including RAF-DB, AffectNet-7, AffectNet-8, SFEW 2.0, FER-2013, and FED-RO, with accuracies of 89.25%, 64.54%, 61.74%, 61.17%, 74.48%, and 71.75%, respectively. All codes and training logs are publicly available at https://github.com/liuhw01/AMP-Net.

Facial Expression Recognition Using Local Sliding Window Attention

“…In existing works, RAN [ 17 ] and AMP-Net [ 16 ] use image-level cropping, but the sizes and positions of the eyes from different facial images in the dataset are very different, so it was difficult for us to choose a suitable local region size to cover all of the important features. In this study, we adopted a feature-level cropping strategy.…”

“…Zhao et al [ 19 ] reduced the occlusion and pose interference through the use of multiscale features and local attention modules. Liu et al [ 16 ] proposed adaptive local cropping, and particularly cropped the eye and mouth parts, guiding the model to find more distinguishable parts. This method is robust to occlusion and pose changes.…”

“…Therefore, researchers have proposed diverse real-world facial expression datasets such as FER2013 [ 10 ], RAF-DB [ 11 ], AffectNet [ 12 ] to advance the development of FER in the wild. The authors of [ 13 , 14 , 15 , 16 ] have made great contributions based on datasets of natural scenes. Li et al [ 15 ] used the face as a whole to obtain various global features to produce a competitive performance, but they ignored the importance of local features.…”

“…Most current methods for obtaining local features crop essential areas of the face through prior knowledge [ 16 , 17 , 18 ] or directly crop the face into patches of fixed size [ 14 , 19 ]. However, these methods have some problems: (1) The method of cropping faces through prior knowledge is in the data preprocessing stage, and most research methods conduct cropping based on landmarks.…”

“…Some studies [ 14 , 16 , 19 ] have obtained multiple local features and directly used them for FER; however, some useless features are learned because some local features contain background and noise information, which is not conducive to FER. Therefore, we enhance the local features to obtain multiscale fine-grained feature information after obtaining multiple local features.…”