Joint Deep Learning of Facial Expression Synthesis and Recognition

Abstract

Recently, deep learning based facial expression recognition (FER) methods have attracted considerable attention and they usually require large-scale labelled training data. Nonetheless, the publicly available facial expression databases typically contain a small amount of labelled data. In this paper, to overcome the above issue, we propose a novel joint deep learning of facial expression synthesis and recognition method for effective FER. More specifically, the proposed method involves a two-stage learning procedure. Firstly, a facial expression synthesis generative adversarial network (FESGAN) is pretrained to generate facial images with different facial expressions. To increase the diversity of the training images, FESGAN is elaborately designed to generate images with new identities from a prior distribution. Secondly, an expression recognition network is jointly learned with the pre-trained FESGAN in a unified framework. In particular, the classification loss computed from the recognition network is used to simultaneously optimize the performance of both the recognition network and the generator of FESGAN. Moreover, in order to alleviate the problem of data bias between the real images and the synthetic images, we propose an intra-class loss with a novel real data-guided back-propagation (RDBP) algorithm to reduce the intra-class variations of images from the same class, which can significantly improve the final performance. Extensive experimental results on public facial expression databases demonstrate the superiority of the proposed method compared with several state-of-the-art FER methods.Index Terms-Facial expression recognition, facial expression synthesis, convolutional neural networks (CNNs), generative adversarial net (GAN).

CFNet: A Coarse-to-Fine Learning Strategy for Facial Expression Recognition

“…Facial expression constitutes a major communication channel of emotion in social interactions [18]- [21]. Therefore, FER also plays a critical role in Computer Vision [22], [23] for emotion understanding. FER approaches can be roughly divided into two categories based on the learning depth, i.e., shallow learning based and deep learning based.…”

Multi-Feature Integrated Concurrent Neural Network for Human Facial Expression Recognition

“…This network provides the 78% classification rate on EIDB-13 intensity-based facial expression dataset and 88% of accuracy rate on the RAF-DB public macro expression dataset. A novel joint deep learning of facial expression synthesis and recognition method is developed in [23] for effective face expression recognition where the facial expression synthesis generative adversarial network (FESGAN) is pre-trained for creating face images with various emotions then expression recognition network is jointly learned with the pre-trained FESGAN. The classification loss is computed to optimize both performances of recognition and generator of FESGAN.…”

Adaptive Deep Disturbance-Disentangled Learning for Facial Expression Recognition

“…During the past few decades, facial expression recognition (FER) has attracted increasing attention in computer vision due to its variety of applications in entertainment, sociable robotics, data-driven animation, and so on (Zhang et al, 2018a,b). Recently, with the considerable development of deep learning, FER has made substantial progress (Chang et al, 2019;Chen et al, 2020;Dapogny et al, 2018;Kollias et al, 2020a;Li and Deng, 2019;Li et al, 2017;Meng et al, 2017;Yan et al, 2020;Yang et al, 2018a;Zhang et al, 2018c).…”