Facial expression and attributes recognition based on multi-task learning of lightweight neural networks

Abstract

In this paper, we describe the results of the HSEmotion team in two tasks of the seventh Affective Behavior Analysis in-thewild (ABAW) competition, namely, multi-task learning for simultaneous prediction of facial expression, valence, arousal, and detection of action units, and compound expression recognition. We propose an efficient pipeline based on frame-level facial feature extractors pre-trained in multi-task settings to estimate valence-arousal and basic facial expressions given a facial photo. We ensure the privacy-awareness of our techniques by using the lightweight architectures of neural networks, such as MT-EmotiDDAMFN, MT-EmotiEffNet, and MT-EmotiMobileFaceNet, that can run even on a mobile device without the need to send facial video to a remote server. It was demonstrated that a significant step in improving the overall accuracy is the smoothing of neural network output scores using Gaussian or box filters. It was experimentally demonstrated that such a simple post-processing of predictions from simple blending of two top visual models improves the F1-score of facial expression recognition up to 7%. At the same time, the mean Concordance Correlation Coefficient (CCC) of valence and arousal is increased by up to 1.25 times compared to each model's frame-level predictions. As a result, our final performance score on the validation set from the multi-task learning challenge is 4.5 times higher than the baseline (1.494 vs 0.32).

Mood Prediction Based on Calendar Events Using Multitask Learning

“…In the application of multitask learning to the prediction of internal states, including mood prediction, the goal is to improve generalization performance by setting multiple objective variables [34], [35]. Li et al [36] proposed a multitask learning method for the problem of predicting mood using physiological signals from wearable sensors as input; they added stress and physical condition variables in addition to mood to the objective variable, leading to an improvement in generalization performance.…”

HSE-NN Team at the 4th ABAW Competition: Multi-task Emotion Recognition and Learning from Synthetic Images

“…Let us consider the details of this pipeline. Its main part, namely, the MT-EmotiEffNet model, was pre-trained using PyTorch framework identically to EfficientNet-B0 from [18] on face identification task. The facial regions in the training set are simply cropped by a face detector without margins or face alignment.…”

“…Thus, every input X and reference X n image is resized to 224x224 and fed into our CNN. We examine two types of features: (1) facial image embeddings (output of the penultimate layer) [18]; and (2) logits (predictions of emotional unnormalized probabilities at the output of the last layer). The outputs of penultimate layer [18] are stored in the D = 1280-dimensional embeddings x and x n , respectively.…”

“…We examine two types of features: (1) facial image embeddings (output of the penultimate layer) [18]; and (2) logits (predictions of emotional unnormalized probabilities at the output of the last layer). The outputs of penultimate layer [18] are stored in the D = 1280-dimensional embeddings x and x n , respectively. The concatenation of 8 FER logits, Valence, and Arousal at the output of the model are stored in the 10-dimensional logits l and l n .…”

“…Foremost, we noticed that the quality of provided synthetic facial images with a resolution of 112x112 is too low for our model that was trained on photos with a rather large resolution (224x224). Hence, it was proposed to enhance these images by using contemporary super- Since the required set of 6 expressions is a subset of eight emotions from AffectNet, the simplest solution is to ignore the training set of synthetic images, and predict expression for an image X with the pre-trained MT-EmotiEffNet by using only 6 scores (emotional posterior probabilities) s (pre) (X) from the last (softmax) layer [18] that is associated with required categories.…”