Cross-Database Evaluation of Pain Recognition from Facial Video

Othman, Ehsan; Werner, Philipp; Saxen, Frerk; Al-Hamadi, Ayoub; Walter, Steffen

doi:10.1109/ispa.2019.8868562

Cited by 23 publications

(19 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The FF we use include 17 AU intensity outputs of OpenFace: AU1, AU2, AU4, AU5, AU6, AU7, AU9, AU10, AU12, AU14, AU15, AU17, AU20, AU23, AU25, AU26, and AU45. The FF and their change over time can be represented by a Time series Statistics Descriptor [ 8 , 10 , 41 , 42 ]. We obtained a 48-dimensional descriptor per time series.…”

Section: Methodsmentioning

confidence: 99%

“…This section describes how we used transfer learning with a reduced MobileNetV2 (MNV2) [ 43 ] to predict pain intensity. We have selected it, as it performed well for binary pain classification from facial video in [ 10 ]. The proposed architecture is a combination of two architectures: (1) MNV2 using the first 5 inverted residual blocks after pre-training with ImageNet for the RGB X-ITE images [ 10 ], (2) the simple CNN architecture for the RFc prediction images, as shown in the previous subsection.…”

Section: Methodsmentioning

confidence: 99%

“…Each time series was summarized by several statistics of the time series itself and of its first and second derivative, including mean, maximum, range, time of maximum, and others, yielding a 48-dimensional descriptor per time series. Othman et al [ 10 ] showed how both RFc and reduced MNV2 performed good for binary pain classification (no pain and highest phasic heat pain intensity). The reduced MNV2 architecture used transfer learning with the first 5 inverted residual blocks; which worked well due to the use of less parameters, avoiding the overfitting that easily happens with the full MNV2 on the quite small pain database [ 10 ].…”

Section: Introductionmentioning

confidence: 99%

“…Othman et al [ 10 ] showed how both RFc and reduced MNV2 performed good for binary pain classification (no pain and highest phasic heat pain intensity). The reduced MNV2 architecture used transfer learning with the first 5 inverted residual blocks; which worked well due to the use of less parameters, avoiding the overfitting that easily happens with the full MNV2 on the quite small pain database [ 10 ]. In this paper, we utilise these previously mentioned networks and simple convolutional neural network (CNN) to predict phasic pain intensity and no pain from facial video.…”

Section: Introductionmentioning

confidence: 99%

“…In this paper, we utilise these previously mentioned networks and simple convolutional neural network (CNN) to predict phasic pain intensity and no pain from facial video. So this paper advances over [ 10 ] by investigating a more complex problem (from binary to multi-class), an advanced recognition approach, and more experiments.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Automatic vs. Human Recognition of Pain Intensity from Facial Expression on the X-ITE Pain Database

Othman

Werner

Saxen

et al. 2021

Sensors

Self Cite

View full text Add to dashboard Cite

Prior work on automated methods demonstrated that it is possible to recognize pain intensity from frontal faces in videos, while there is an assumption that humans are very adept at this task compared to machines. In this paper, we investigate whether such an assumption is correct by comparing the results achieved by two human observers with the results achieved by a Random Forest classifier (RFc) baseline model (called RFc-BL) and by three proposed automated models. The first proposed model is a Random Forest classifying descriptors of Action Unit (AU) time series; the second is a modified MobileNetV2 CNN classifying face images that combine three points in time; and the third is a custom deep network combining two CNN branches using the same input as for MobileNetV2 plus knowledge of the RFc. We conduct experiments with X-ITE phasic pain database, which comprises videotaped responses to heat and electrical pain stimuli, each of three intensities. Distinguishing these six stimulation types plus no stimulation was the main 7-class classification task for the human observers and automated approaches. Further, we conducted reduced 5-class and 3-class classification experiments, applied Multi-task learning, and a newly suggested sample weighting method. Experimental results show that the pain assessments of the human observers are significantly better than guessing and perform better than the automatic baseline approach (RFc-BL) by about 1%; however, the human performance is quite poor due to the challenge that pain that is ethically allowed to be induced in experimental studies often does not show up in facial reaction. We discovered that downweighting those samples during training improves the performance for all samples. The proposed RFc and two-CNNs models (using the proposed sample weighting) significantly outperformed the human observer by about 6% and 7%, respectively.

show abstract