A Facial Expression Recognition Approach for Social IoT Frameworks

Barra, Silvio; Hossain, Sanoar; Pero, Chiara; Umer, Saiyed

doi:10.1016/j.bdr.2022.100353

Cited by 9 publications

(9 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent studies turned to fusion models of two [5,9,22] or more CNNs [23]. Furthermore, a recent study implemented IoT in the FER system [24].…”

Section: Related Workmentioning

confidence: 99%

“…While most studies focus on improving recognition accuracy, challenges such as poses, occlusion, and illumination demand further investigation and more efficient solutions. In order to compare the SAFEPA with the state-of-the-art, we have selected four recent studies [17,[24][25][26] that use CNN, HOG, and LBP for FER on the same datasets as our study. Additionally, we included three more recent studies that focus on facial expression-based pain assessment [3,22,23].…”

Section: Related Workmentioning

confidence: 99%

“…In [24], Barra and his colleagues developed a facial expression recognition system based on a social IoT solution. To detect the face region, they applied the tree-structured model.…”

Section: Related Workmentioning

confidence: 99%

“…This subset comprised a total of 980 samples. The obtained results were compared with four recently published models for Facial Expression Recognition (FER) mentioned in references [17,[24][25][26].…”

Section: Kdef Dataset-frontal View Subsetmentioning

confidence: 99%

See 3 more Smart Citations

SAFEPA: An Expandable Multi-Pose Facial Expressions Pain Assessment Method

Alghamdi

Alaghband

2023

Applied Sciences

View full text Add to dashboard Cite

Accurately assessing the intensity of pain from facial expressions captured in videos is crucial for effective pain management and critical for a wide range of healthcare applications. However, in uncontrolled environments, detecting facial expressions from full left and right profiles remains a significant challenge, and even the most advanced models for recognizing pain levels based on facial expressions can suffer from declining performance. In this study, we present a novel model designed to overcome the challenges posed by full left and right profiles—Sparse Autoencoders for Facial Expressions-based Pain Assessment (SAFEPA). Our model utilizes Sparse Autoencoders (SAE) to reconstruct the upper part of the face from the input image, and feeds both the original image and the reconstructed upper face into two pre-trained concurrent and coupled Convolutional Neural Networks (CNNs). This approach gives more weight to the upper part of the face, resulting in superior recognition performance. Moreover, SAFEPA’s design leverages CNNs’ strengths while also accommodating variations in head poses, thus eliminating the need for face detection and upper-face extraction preprocessing steps needed in other models. SAFEPA achieves high accuracy in recognizing four levels of pain on the widely used UNBC-McMaster shoulder pain expression archive dataset. SAFEPA is extended for facial expression recognition, where we show it to outperform state-of-the-art models in recognizing seven facial expressions viewed from five different angles, including the challenging full left and right profiles, on the Karolinska Directed Emotional Faces (KDEF) dataset. Furthermore, the SAFEPA system is capable of processing BioVid Heat Pain datasets with an average processing time of 17.82 s per video (5 s in length), while maintaining a competitive accuracy compared to other state-of-the-art pain detection systems. This experiment demonstrates its applicability in real-life scenarios for monitoring systems. With SAFEPA, we have opened new possibilities for accurate pain assessment, even in challenging situations with varying head poses.

show abstract

“…Recent studies turned to fusion models of two [5,9,22] or more CNNs [23]. Furthermore, a recent study implemented IoT in the FER system [24].…”

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

“…In [24], Barra and his colleagues developed a facial expression recognition system based on a social IoT solution. To detect the face region, they applied the tree-structured model.…”

Section: Related Workmentioning

confidence: 99%

Section: Kdef Dataset-frontal View Subsetmentioning

confidence: 99%

See 2 more Smart Citations

SAFEPA: An Expandable Multi-Pose Facial Expressions Pain Assessment Method

Alghamdi

Alaghband

2023

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…With the rapid development of Internet of Things (IoT) technology, the application scenarios of emotion recognition have been further expanded. [17][18][19] By using embedded multiple sensors and wireless data transmission networks, it is easier to collect multimodal emotional information, which is shown in Figure 1. Facial expression recognition (FER) based on multi-modality has received more and more attention.…”

Section: Introductionmentioning

confidence: 99%

A joint hierarchical cross‐attention graph convolutional network for multi‐modal facial expression recognition

Xu,

Du,

Wang

et al. 2023

Computational Intelligence

View full text Add to dashboard Cite

Emotional recognition in conversations (ERC) is increasingly being applied in various IoT devices. Deep learning‐based multimodal ERC has achieved great success by leveraging diverse and complementary modalities. Although most existing methods try to adopt attention mechanisms to fuse different information, these methods ignore the complementarity between modalities. To this end, the joint cross‐attention model is introduced to alleviate this issue. However, multi‐scale feature information on different modalities is not utilized. Moreover, the context relationship plays an important role in feature extraction in the expression recognition task. In this paper, we propose a novel joint hierarchical graph convolution network (JHGCN) which exploits different layer features and context relationships for facial expression recognition based on audio‐visual (A‐V) information. Specifically, we adopt different deep networks to extract features from different modalities individually. For V modality, we construct V graph data based on patch embeddings which are extracted from the transformer encoder. Moreover, we embed the graph convolution which can leverage the intra‐modality relationships with the transformer encoder. Then, the deep feature from different layers is fed to the hierarchical fusion module to enhance feature representation. At last, we use the joint cross‐attention mechanism to exploit the complementary inter‐modality relationships. To validate the proposed model, we have conducted various experiments on the AffWild2 and CMU‐MOSI datasets. All results confirm that our proposed model achieves highly promising performance compared to the joint cross‐attention model and other methods.

show abstract