Two-Stream Attention Network for Pain Recognition from Video Sequences

Abstract

Several approaches have been proposed for the analysis of pain-related facial expressions. These approaches range from common classification architectures based on a set of carefully designed handcrafted features, to deep neural networks characterised by an autonomous extraction of relevant facial descriptors and simultaneous optimisation of a classification architecture. In the current work, an end-to-end approach based on attention networks for the analysis and recognition of pain-related facial expressions is proposed. The method combines both spatial and temporal aspects of facial expressions through a weighted aggregation of attention-based neural networks’ outputs, based on sequences of Motion History Images (MHIs) and Optical Flow Images (OFIs). Each input stream is fed into a specific attention network consisting of a Convolutional Neural Network (CNN) coupled to a Bidirectional Long Short-Term Memory (BiLSTM) Recurrent Neural Network (RNN). An attention mechanism generates a single weighted representation of each input stream (MHI sequence and OFI sequence), which is subsequently used to perform specific classification tasks. Simultaneously, a weighted aggregation of the classification scores specific to each input stream is performed to generate a final classification output. The assessment conducted on both the BioVid Heat Pain Database (Part A) and SenseEmotion Database points at the relevance of the proposed approach, as its classification performance is on par with state-of-the-art classification approaches proposed in the literature.

Emotion and Stress Recognition Related Sensors and Machine Learning Technologies

“…In [ 19 ], the authors revisited the analysis of pain-related facial expressions by proposing an end-to-end approach based on attention networks for the analysis and recognition of pain-related facial expressions. The method proposed by the authors combined both spatial and temporal aspects of facial expressions through a weighted aggregation of attention-based neural networks’ outputs that use sequences of Motion History Images (MHIs) and Optical Flow Images (OFIs).…”

Basketball-51: A Video Dataset for Activity Recognition in the Basketball Game

“…Many machines learning and deep learning architecture has been used for action recognition using these video datasets. [25] which uses two different types of stream, RGB and flow data, are fused for prediction. For our analysis, we used the 3D ConvNet to train our model from scratch for both our RGB and optical flow video data along with two-stream 3D ConvNet with early fusion techniques.…”

Emotion and Stress Recognition Related Sensors and Machine Learning Technologies

“…In [19], the authors revisited the analysis of pain-related facial expressions by proposing an end-to-end approach based on attention networks for the analysis and recognition of pain-related facial expressions. The method proposed by the authors combined both spatial and temporal aspects of facial expressions through a weighted aggregation of attentionbased neural networks' outputs that use sequences of Motion History Images (MHIs) and Optical Flow Images (OFIs).…”