No abstract

Facial emotion recognition system (FERS) recognize the person’s emotions based on various image processing stages including feature extraction as one of the major processing steps. In this study, we presented a hybrid approach for recognizing facial expressions by performing the feature level fusion of a local and a global feature descriptor that is classified by a support vector machine (SVM) classifier. Histogram of oriented gradients (HoG) is selected for the extraction of global facial features and local intensity order pattern (LIOP) to extract the local features. As HoG is a shape-based descriptor, with the help of edge information, it can extract the deformations caused in facial muscles due to changing emotions. On the contrary, LIOP works based on the information of pixels intensity order and is invariant to change in image viewpoint, illumination conditions, JPEG compression, and image blurring as well. Thus both the descriptors proved useful to recognize the emotions effectively in the images captured in both constrained and realistic scenarios. The performance of the proposed model is evaluated based on the lab-constrained datasets including CK+, TFEID, JAFFE as well as on realistic datasets including SFEW, RaF, and FER-2013 dataset. The optimal recognition accuracy of 99.8%, 98.2%, 93.5%, 78.1%, 63.0%, 56.0% achieved respectively for CK+, JAFFE, TFEID, RaF, FER-2013 and SFEW datasets respectively.

This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

A picture is worth a thousand words to convey information during daily life communication. Recognizing a person’s emotions from facial expressions (FEs) has become a prevalent research field in the past few decades. Deep learning (DL) models, mainly deep convolutional neural networks (DCNNs), are trending in the last few years to perform recognition/classification tasks. Several prior DCNNs have resulted in good recognition accuracy for facial emotion recognition (FER) systems. Still, there is a need for an effective as well as efficient FER system that can recognize the FEs irrespective of the illumination conditions, subjects’ gender, age range, geographical locations, race, etc. In this work, we have presented a novel lightweight DCNN model that can recognize the FEs in the aforementioned conditions. We have made it lightweight by optimal selection of its hidden layers, which ultimately resulted in the reduced number of floating point operations (FLOPs). We have embedded the data augmentation step into the model’s training phase to enhance its generalization ability. An early stopping criterion is also introduced to prevent the model’s overfitting. We have trained and evaluated the performance of the proposed model on widely used benchmarks for FEs databases. We have selected five diverse databases (two collected in the lab, one based on stylized cartoon characters, and two collections in an unconstrained realistic environment), including CK+, Karolinska Directed Emotional Faces (KDEF), Facial Expression Research Group (FERG), Facial Expression Recognition-2013 (FER-2013), and Real-World Affective Faces Database (RaF-DB). The recognition accuracy of 99.98%, 99.25%, 88.17%, 84.09%, 69.87%, and 69.16% is achieved for the FERG, CK+, KDEF, RaF-DB, FER-2013 human faces (FER-2013H), and FER-2013 complete (FER-2013C) database, respectively. Our model outperforms the number of state-of-the-art FER approaches in terms of recognition accuracy and FLOPs.