Facial expression and attributes recognition based on multi-task learning of lightweight neural networks

Abstract

In this paper, we examine the multi-task training of lightweight convolutional neural networks for face identification and classification of facial attributes (age, gender, ethnicity) trained on cropped faces without margins. It is shown that it is still necessary to fine-tune these networks in order to predict facial expressions. Several models are presented based on MobileNet, EfficientNet and RexNet architectures. It was experimentally demonstrated that our models are characterized by the state-of-the-art emotion classification accuracy on AffectNet dataset and near state-ofthe-art results in age, gender and race recognition for UTKFace dataset. Moreover, it is shown that the usage of our neural network as a feature extractor of facial regions in video frames and concatenation of several statistical functions (mean, max, etc.) leads to 4.5% higher accuracy than the previously known state-of-the-art single models for AFEW and VGAF datasets from the EmotiW challenges. The models and source code are publicly available at https://github.com/HSE-asavchenko/ face-emotion-recognition.

Distract Your Attention: Multi-head Cross Attention Network for Facial Expression Recognition

“…PhaNet (Liu et al 2019) 54.82 ESR-9 (Siqueira et al 2020) 59.30 RAN (Wang et al 2020b) 59.50 SCN (Wang et al 2020a) 60.23 PSR (Vo et al 2020) 60.68 EfficientFace (Zhao et al 2021) 59.89 EfficientNet-B0 (Savchenko 2021) 61.32 MViT (Li et (Chen et al 2019) 61.25 LDL-ALSG (Chen et al 2020) 59.35 VGG-FACE (Kollias et al 2020) 60.00 OADN (Ding et al 2020) 61.89 DDA-Loss (Farzaneh et al 2020) 62.34 EfficientFace (Zhao et al 2021) 63.70 MViT (Li et al 2021) 64.57…”

Comparing Facial Expression Recognition in Humans and Machines: Using CAM, GradCAM, and Extremal Perturbation

“…However, FER models perform less well with in-the-wild datasets: the stateof-the-art (SOTA) accuracy on the FER+ dataset with cleaned and updated labels is 89.75% reached by a PSR model on seven expressions [29]. SOTA on the AffectNet dataset, in contrast, is only 65.74% for seven of the eight included emotion categories [25]. Given that FER systems in practical use, such as humanoid robots and surveillance systems, will not be fed with regular illuminations, frontal head position and exemplary expressions, it is important to improve FER accuracy on such in-the-wild datasets.…”

“…Automatic FER has come a long way, from hand-crafted approaches to the current end-to-end deep learning models that locate and recognize facial expressions [16]. Nonetheless, there still is a long way to go: current algorithms are good enough at recognizing laboratory-controlled facial expression images, but they struggle to recognize expressions from naturalistic images [18,25].…”

Analytical Framework for Facial Expression on Game Experience Test

“…Paul Ekman and Wallace V. Friesen et al proposed a method for using facial muscle states to associate facial expression with six basic emotions: happiness, sadness, anger, disgust, surprise, and fear [21]. Recently, deep learning technology has been used, such as that in Savchenko, Andrey V et al [22].…”