Multimodal Data Fusion for Person-Independent, Continuous Estimation of Pain Intensity

Kächele, Markus; Thiam, Patrick; Amirian, Mohammadreza; Werner, Philipp; Walter, Steffen; Schwenker, Friedhelm; Palm, Günther

doi:10.1007/978-3-319-23983-5_26

Cited by 59 publications

(37 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The metrics used to quantify the performance of automatic pain detection from facial expressions depend on the learning task. For classification tasks, metrics such as accuracy, F1 score, and area under Receiver Operating Rudovic et al [121] hidden conditional random field Lopez-Martinez et al [118] regularized multi-task learning Romera-Paredes et al [108] two-step SVM step1-AU: Lucey et al [83], Lucey et al [128] step2-pain: Bartlett et al [131] both steps: Littlewort et al [72], Littlewort et al [56], Ghasemi et al [74] logistical linear regression step2-pain: Lucey et al [83], Lucey et al [128] k-nearest neighbors step1-AU: Zafar and Khan [129] logistic regression step2-pain: Sikka et al [59] alignment-based learning step2-pain: Schmid et al [77], Siebers et al [78] hidden conditional random field step2-pain: Ghasemi et al [74] latent-dynamic conditional random field step1-AU: Zhang et al [76] regression one-step support vector regression Florea et al [111], Lopez-Martinez et al [118] ordinal support vector regression Zhao et al [114] relevance vector regression or its variants Kaltwang et al [107], Kaltwang et al [113], Egede et al [116], Egede and Valstar [117] random forest Kächele et al [103] linear regression Neshov and Manolova [94] ordinal support vector regression Zhao et al [114] NN Lopez-Martinez et al [118] Convolutional Neural Network (CNN) Wang et al [109] 3D CNN with kernels of varying temporal lengths Tavakolian and Hadid [119] recurrent CNN Zhou et al [112] LSTM recurrent neural network Rodriguez et al [115], Lopez-Martinez et al [118] two-step support vector regression step1-AU: B...…”

Section: Learning Methodsmentioning

confidence: 99%

Automatic Detection of Pain from Facial Expressions: A Survey

Hassan

Seus

Wollenberg

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Pain sensation is essential for survival, since it draws attention to physical threat to the body. Pain assessment is usually done through self-reports. However, self-assessment of pain is not available in the case of noncommunicative patients, and therefore, observer reports should be relied upon. Observer reports of pain could be prone to errors due to subjective biases of observers. Moreover, continuous monitoring by humans is impractical. Therefore, automatic pain detection technology could be deployed to assist human caregivers and complement their service, thereby improving the quality of pain management, especially for noncommunicative patients. Facial expressions are a reliable indicator of pain, and are used in all observer-based pain assessment tools. Following the advancements in automatic facial expression analysis, computer vision researchers have tried to use this technology for developing approaches for automatically detecting pain from facial expressions. This paper surveys the literature published in this field over the past decade, categorizes it, and identifies future research directions. The survey covers the pain datasets used in the reviewed literature, the learning tasks targeted by the approaches, the features extracted from images and image sequences to represent pain-related information, and finally, the machine learning methods used.

show abstract

Section: Learning Methodsmentioning

confidence: 99%

Automatic Detection of Pain from Facial Expressions: A Survey

Hassan

Seus

Wollenberg

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…The assessed spatial descriptors consist of Local Binary Patterns (LBP) [28], Local Phase Quantization (LPQ) [29], Binarized Statistical Image Features (BSIF) [30] as well as each descriptor's spatio-temporal counterpart extracted from video sequences on three orthogonal planes (LBP-TOP, LPQ-TOP and BSIF-TOP). In [8,31], the authors propose several sets of spatio-temporal facial action descriptors based on both appearance-and geometry-based features extracted from both the facial area, as well as the head pose. Those descriptors are further used to perform the classification of several levels of pain intensities using a Random Forest (RF) [32] model.…”

Section: Related Workmentioning

confidence: 99%

Two-Stream Attention Network for Pain Recognition from Video Sequences

Thiam

Kestler

Schwenker

2020

Sensors

Self Cite

View full text Add to dashboard Cite

Several approaches have been proposed for the analysis of pain-related facial expressions. These approaches range from common classification architectures based on a set of carefully designed handcrafted features, to deep neural networks characterised by an autonomous extraction of relevant facial descriptors and simultaneous optimisation of a classification architecture. In the current work, an end-to-end approach based on attention networks for the analysis and recognition of pain-related facial expressions is proposed. The method combines both spatial and temporal aspects of facial expressions through a weighted aggregation of attention-based neural networks’ outputs, based on sequences of Motion History Images (MHIs) and Optical Flow Images (OFIs). Each input stream is fed into a specific attention network consisting of a Convolutional Neural Network (CNN) coupled to a Bidirectional Long Short-Term Memory (BiLSTM) Recurrent Neural Network (RNN). An attention mechanism generates a single weighted representation of each input stream (MHI sequence and OFI sequence), which is subsequently used to perform specific classification tasks. Simultaneously, a weighted aggregation of the classification scores specific to each input stream is performed to generate a final classification output. The assessment conducted on both the BioVid Heat Pain Database (Part A) and SenseEmotion Database points at the relevance of the proposed approach, as its classification performance is on par with state-of-the-art classification approaches proposed in the literature.

show abstract

“…Gruss et al [8] also applied SVMs to the same dataset, but only used EMG, ECG and SC features. These features have also been used in combination with behavioral features derived from video [24], [25], [26]. Recently, Kchele et al [5] proposed a method for personalized prediction of pain intensity based on similarity measures, using EMG, ECG and SC features, together with meta-information.…”

Section: Related Workmentioning

confidence: 99%

Multi-task neural networks for personalized pain recognition from physiological signals

Lopez-Martinez

Picard

2017

2017 Seventh International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)

View full text Add to dashboard Cite

Pain is a complex and subjective experience that poses a number of measurement challenges. While self-report by the patient is viewed as the gold standard of pain assessment, this approach fails when patients cannot verbally communicate pain intensity or lack normal mental abilities. Here, we present a pain intensity measurement method based on physiological signals. Specifically, we implement a multi-task learning approach based on neural networks that accounts for individual differences in pain responses while still leveraging data from across the population. We test our method in a dataset containing multi-modal physiological responses to nociceptive pain.

show abstract

Multimodal Data Fusion for Person-Independent, Continuous Estimation of Pain Intensity

Cited by 59 publications

References 15 publications

Automatic Detection of Pain from Facial Expressions: A Survey

Automatic Detection of Pain from Facial Expressions: A Survey

Two-Stream Attention Network for Pain Recognition from Video Sequences

Multi-task neural networks for personalized pain recognition from physiological signals

Contact Info

Product

Resources

About