Spatio-temporal Pain Recognition in CNN-Based Super-Resolved Facial Images

Abstract

Automatic pain detection is a long expected solution to a prevalent medical problem of pain management. This is more relevant when the subject of pain is young children or patients with limited ability to communicate about their pain experience. Computer vision-based analysis of facial pain expression provides a way of efficient pain detection. When deep machine learning methods came into the scene, automatic pain detection exhibited even better performance. In this paper, we figured out three important factors to exploit in automatic pain detection: spatial information available regarding to pain in each of the facial video frames, temporal axis information regarding to pain expression pattern in a subject video sequence, and variation of face resolution. We employed a combination of convolutional neural network and recurrent neural network to setup a deep hybrid pain detection framework that is able to exploit both spatial and temporal pain information from facial video. In order to analyze the effect of different facial resolutions, we introduce a super-resolution algorithm to generate facial video frames with different resolution setups. We investigated the performance on the publicly available UNBC-McMaster Shoulder Pain database. As a contribution, the paper provides novel and important information regarding to the performance of a hybrid deep learning framework for pain detection in facial images of different resolution.

Rehabilitation of Traumatic Brain Injured Patients: Patient Mood Analysis from Multimodal Video

“…using deep learning techniques show that the two employed modalities can complement each other on classifying patients status to positive or negative. In terms of methodology, contributions by [12] and [13] are probably most close to our method but these systems work well for healthy people in controlled environment. Moreover these systems have luxury of data sets where subjects were cooperative with no or less pose variation, minimum occlusions and high quality images unlike with TBI patients.…”

Changes in Facial Expression as Biometric: A Database and Benchmarks of Identification

“…Spatial information comes from facial features in a single video frame. On the other hand, temporal information stands for the relationship between facial features revealed in consecutive video frames [2], [9], [22]. CNNs are well known for their great ability in learning abstract spatial features from a given image (single frame) [11].…”

Multi-Channel Neural Network for Assessing Neonatal Pain from Videos

“…The prediction for each of these modalities was obtained using binary classifiers (e.g., SVM) trained using handcrafted features. Recently, deep learning methods [15], [16] have become popular in pain classification. In case of infants, Celona and Manoni [17] proposed a framework that combines both handcrafted and deep features for classifying COPE images as pain/no-pain.…”