SAFEPA: An Expandable Multi-Pose Facial Expressions Pain Assessment Method

Abstract

Accurately assessing the intensity of pain from facial expressions captured in videos is crucial for effective pain management and critical for a wide range of healthcare applications. However, in uncontrolled environments, detecting facial expressions from full left and right profiles remains a significant challenge, and even the most advanced models for recognizing pain levels based on facial expressions can suffer from declining performance. In this study, we present a novel model designed to overcome the challenges posed by full left and right profiles—Sparse Autoencoders for Facial Expressions-based Pain Assessment (SAFEPA). Our model utilizes Sparse Autoencoders (SAE) to reconstruct the upper part of the face from the input image, and feeds both the original image and the reconstructed upper face into two pre-trained concurrent and coupled Convolutional Neural Networks (CNNs). This approach gives more weight to the upper part of the face, resulting in superior recognition performance. Moreover, SAFEPA’s design leverages CNNs’ strengths while also accommodating variations in head poses, thus eliminating the need for face detection and upper-face extraction preprocessing steps needed in other models. SAFEPA achieves high accuracy in recognizing four levels of pain on the widely used UNBC-McMaster shoulder pain expression archive dataset. SAFEPA is extended for facial expression recognition, where we show it to outperform state-of-the-art models in recognizing seven facial expressions viewed from five different angles, including the challenging full left and right profiles, on the Karolinska Directed Emotional Faces (KDEF) dataset. Furthermore, the SAFEPA system is capable of processing BioVid Heat Pain datasets with an average processing time of 17.82 s per video (5 s in length), while maintaining a competitive accuracy compared to other state-of-the-art pain detection systems. This experiment demonstrates its applicability in real-life scenarios for monitoring systems. With SAFEPA, we have opened new possibilities for accurate pain assessment, even in challenging situations with varying head poses.

A Review of Automatic Pain Assessment from Facial Information Using Machine Learning

“…The availability of many face images and/or video datasets for pain assessment has driven recent advances in the field of automatic pain assessment. The UNBC-McMaster Shoulder Pain Expression Archive Database (UNBC-McMaster) [44] is the one of the most widely utilized of these datasets [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][17][18][19][20][21]. This dataset was gathered from 25 adult participants suffering from shoulder pain, which form 48,398 RGB frames issued from 200 variable-length videos (see details in Table 1).…”

“…As a result, the UNBC-McMaster dataset size was reduced from 48,398 frames to 10,783 frames. Furthermore, another well-used dataset [6,8,20,45,59] is the BioVid Heat Pain dataset [45,59], which was formed by collecting 17300 RGB videos from 87 subjects of 5 s each with a frame rate of 25 fps. An inducted heat pain was the cause of the pain represented in these videos.…”

“…However, deep learning-based methods [5][6][7][8][9][10][11][12][13] use deep learning models (e.g., Convolutional Neural Networks (CNNs) [49], Residual neural network (ResNet) [50], VGG [51] and Inception Network [52]) for both feature extraction and model-learning tasks. Recently, hybrid modelbased methods [14][15][16][17][18][19][20][21], which combine machine learning and/or deep learning have been proposed for the pain assessment task. These methods follow different ensemble learning strategies to leverage the advantages of different learning models.…”

Determinants of Emotion Recognition System Adoption: Empirical Evidence from Malaysia

“…The experiments concluded that the implementation of facial recognition as part of ERS towards video surveillance system can improve the reliability of abnormal behavior detection via facial expressions depending on different emotions and environmental conditions. Furthermore, facial expressions can be used in identifying pain, which will benefit the healthcare industry [39]. Assessing a patient's pain levels over time is deemed to be important, specifically regarding the effectiveness of medical treatments.…”

Ensemble Learning-Based Pain Intensity Identification Model Using Facial Expressions

“…Previous investigations have demonstrated that facial action units (AUs) may be detected for autonomous pain expression evaluation (Alghamdi and Alaghband, 2023). On the contrary, existing methods for identifying facial pain AUs are restricted to regulated circumstances.…”