Transformer-Based Self-Supervised Learning for Emotion Recognition

Abstract

In order to exploit representations of time-series signals, such as physiological signals, it is essential that these representations capture relevant information from the whole signal. In this work, we propose to use a Transformer-based model to process electrocardiograms (ECG) for emotion recognition. Attention mechanisms of the Transformer can be used to build contextualized representations for a signal, giving more importance to relevant parts. These representations may then be processed with a fully-connected network to predict emotions.To overcome the relatively small size of datasets with emotional labels, we employ self-supervised learning. We gathered several ECG datasets with no labels of emotion to pre-train our model, which we then fine-tuned for emotion recognition on the AMIGOS dataset. We show that our approach reaches state-of-the-art performances for emotion recognition using ECG signals on AMIGOS. More generally, our experiments show that transformers and pre-training are promising strategies for emotion recognition with physiological signals.

Applying Self-Supervised Representation Learning for Emotion Recognition Using Physiological Signals

“…Rodriguez et al [ 91 ] proposed a transform-based model to process ECG signals, in which this mechanism is used to build contextualized representations of the signal, which give more importance to the relevant parts to predict emotions. The authors employed self-supervised learning to solve the problem with a small amount of labeled data.…”

Emotion Recognition with Pre-Trained Transformers Using Multimodal Signals

“…For our ECG emotion recognizer, we employ the approach described in [15] and depicted in Figure 1. In this approach, a pre-training step is first used prior to fine-tuning the model to improve its performance.…”

“…The ECG single-modality emotion recognition model from [15] is based on the Transformer [10]. The Transformer is an architecture capable of incorporating contextualized information thanks to its self-attention mechanisms.…”

“…To build our EEG single-modality emotion recognition model, we adapt the ECG model from [15] and described in the previous subsection, to accept EEG signals.…”

“…Another advantage of using a Transformer is that we can benefit from a very successful pre-training technique described in BERT [12] and developed for NLP tasks, which we can adapt to our needs. This pre-training strategy has been successfully adapted to other domains like Computer Vision [13], Speech Processing [14] and Affective Computing [15].…”