Emotion Detection Using Facial Expression Involving Occlusions and Tilt

Qazi, Awais Salman; Farooq, Muhammad Shoaib; Rustam, Furqan; Villar, Mónica Gracia; Rodríguez, Carmen Lili; Ashraf, Imran

doi:10.3390/app122211797

Cited by 12 publications

(4 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This contribution makes the model very applicable, for example, in the work of Cong, G. et al [29], facial expressions are used in dubbing tasks; if our research were applied in this case, the speaker's expression could be recognized and applied in real time. The advantage of this approach is that it is scalable to various fields, such as environments including occlusion and tilt, or for low-resolution facial expression recognition [30,31]. To accomplish this, we tried a variety of methods.…”

Section: Discussionmentioning

confidence: 99%

Fast and Accurate Facial Expression Image Classification and Regression Method Based on Knowledge Distillation

2023

View full text Add to dashboard Cite

As emotional states are diverse, simply classifying them through discrete facial expressions has its limitations. Therefore, to create a facial expression recognition system for practical applications, not only must facial expressions be classified, emotional changes must be measured as continuous values. Based on the knowledge distillation structure and the teacher-bounded loss function, we propose a method to maximize the synergistic effect of jointly learning discrete and continuous emotional states of eight expression classes, valences, and arousal levels. The proposed knowledge distillation model uses Emonet, a state-of-the-art continuous estimation method, as the teacher model, and uses a lightweight network as the student model. It was confirmed that performance degradation can be minimized even though student models have multiply-accumulate operations of approximately 3.9 G and 0.3 G when using EfficientFormer and MobileNetV2, respectively, which is much less than the amount of computation required by the teacher model (16.99 G). Together with the significant improvements in computational efficiency (by 4.35 and 56.63 times using EfficientFormer and MobileNetV2, respectively), the decreases in facial expression classification accuracy were approximately 1.35% and 1.64%, respectively. Therefore, the proposed method is optimized for application-level interaction systems in terms of both the amount of computation required and the accuracy.

show abstract

Section: Discussionmentioning

confidence: 99%

Fast and Accurate Facial Expression Image Classification and Regression Method Based on Knowledge Distillation

2023

View full text Add to dashboard Cite

show abstract

“…commonly applied to analyzing visual imagery. They have been highly successful in various tasks in computer vision, including image and video recognition, image classification, medical image analysis, and many more for the paper in [11]. CNNs are outlined to naturally and adaptively learn spatial pecking orders of highlights from input pictures.…”

Section: Data Sectionmentioning

confidence: 99%

Improving Facial Expression Classification through Ensemble Deep Learning Models

2024

nano-ntp

View full text Add to dashboard Cite

This article presents a comprehensive study on improving facial expression classification through the deployment of an ensemble deep learning model that amalgamates multiple advanced CNN architectures, including Inception V3 (Inception Architecture Version-3), ResNet50 (Residual Network with 50 layers), and SPP-net (Spatial Pyramid Pooling in Deep Convolutional Networks). By leveraging the strengths of these diverse models, the ensemble approach aims to capture a richer representation of facial features, thereby enhancing classification accuracy. The study meticulously outlines the data preprocessing techniques employed, such as image segmentation and normalization, to prepare the AffectNet dataset for model training and evaluation. On the AffectNet dataset, the proposed ensemble model achieves 85.2% classification accuracy, outperforming the best individual CNN by 5-10%. This marks a significant improvement, showcasing the potential of ensemble methods in the field of facial expression recognition. The article highlights future research directions, including the exploration of end-toend trainable ensemble models and the collection of more diverse datasets to further refine and enhance model performance. This study contributes to the ongoing advancements in affective computing by demonstrating the effectiveness of ensemble deep-learning models in improving facial expression classification accuracy. It opens new avenues for research and application, promising to enrich human-computer interactions and deepen our understanding of human emotions.

show abstract

“…Where accuracy shows the gap between the real values and the predicted values, precision deals with the fraction of positive predictions as in Equation ( 7) and recall deals with the actual positive fraction as in Equation ( 8). Precision = TP TP + FP (7) Recall = TP TP + FN (8) The F1-Score combines both precision and recall in a single value as in Equation ( 9). The F1-Score is indicative of a model's balanced ability to describe both positive cases (recall) as well as be accurate with the cases that it captures (precision).…”

Section: The Evaluationmentioning

confidence: 99%

“…Developing such systems is no easy task due to a variety of challenges, including variations in head poses, illumination [7], and occlusion [8]. Despite significant progress in recent years, many deep-learning-based pain assessment models rely on face detection algorithms as a critical data processing step, which may not work accurately for full left/right profile views.…”

Section: Introductionmentioning

confidence: 99%

SAFEPA: An Expandable Multi-Pose Facial Expressions Pain Assessment Method

Alghamdi

Alaghband

2023

Applied Sciences

View full text Add to dashboard Cite

Accurately assessing the intensity of pain from facial expressions captured in videos is crucial for effective pain management and critical for a wide range of healthcare applications. However, in uncontrolled environments, detecting facial expressions from full left and right profiles remains a significant challenge, and even the most advanced models for recognizing pain levels based on facial expressions can suffer from declining performance. In this study, we present a novel model designed to overcome the challenges posed by full left and right profiles—Sparse Autoencoders for Facial Expressions-based Pain Assessment (SAFEPA). Our model utilizes Sparse Autoencoders (SAE) to reconstruct the upper part of the face from the input image, and feeds both the original image and the reconstructed upper face into two pre-trained concurrent and coupled Convolutional Neural Networks (CNNs). This approach gives more weight to the upper part of the face, resulting in superior recognition performance. Moreover, SAFEPA’s design leverages CNNs’ strengths while also accommodating variations in head poses, thus eliminating the need for face detection and upper-face extraction preprocessing steps needed in other models. SAFEPA achieves high accuracy in recognizing four levels of pain on the widely used UNBC-McMaster shoulder pain expression archive dataset. SAFEPA is extended for facial expression recognition, where we show it to outperform state-of-the-art models in recognizing seven facial expressions viewed from five different angles, including the challenging full left and right profiles, on the Karolinska Directed Emotional Faces (KDEF) dataset. Furthermore, the SAFEPA system is capable of processing BioVid Heat Pain datasets with an average processing time of 17.82 s per video (5 s in length), while maintaining a competitive accuracy compared to other state-of-the-art pain detection systems. This experiment demonstrates its applicability in real-life scenarios for monitoring systems. With SAFEPA, we have opened new possibilities for accurate pain assessment, even in challenging situations with varying head poses.

show abstract

Emotion Detection Using Facial Expression Involving Occlusions and Tilt

Cited by 12 publications

References 37 publications

Fast and Accurate Facial Expression Image Classification and Regression Method Based on Knowledge Distillation

Fast and Accurate Facial Expression Image Classification and Regression Method Based on Knowledge Distillation

Improving Facial Expression Classification through Ensemble Deep Learning Models

SAFEPA: An Expandable Multi-Pose Facial Expressions Pain Assessment Method

Contact Info

Product

Resources

About