Emotion Assessment Using Feature Fusion and Decision Fusion Classification Based on Physiological Data: Are We There Yet?

Abstract

Emotion recognition based on physiological data classification has been a topic of increasingly growing interest for more than a decade. However, there is a lack of systematic analysis in literature regarding the selection of classifiers to use, sensor modalities, features and range of expected accuracy, just to name a few limitations. In this work, we evaluate emotion in terms of low/high arousal and valence classification through Supervised Learning (SL), Decision Fusion (DF) and Feature Fusion (FF) techniques using multimodal physiological data, namely, Electrocardiography (ECG), Electrodermal Activity (EDA), Respiration (RESP), or Blood Volume Pulse (BVP). The main contribution of our work is a systematic study across five public datasets commonly used in the Emotion Recognition (ER) state-of-the-art, namely: (1) Classification performance analysis of ER benchmarking datasets in the arousal/valence space; (2) Summarising the ranges of the classification accuracy reported across the existing literature; (3) Characterising the results for diverse classifiers, sensor modalities and feature set combinations for ER using accuracy and F1-score; (4) Exploration of an extended feature set for each modality; (5) Systematic analysis of multimodal classification in DF and FF approaches. The experimental results showed that FF is the most competitive technique in terms of classification accuracy and computational complexity. We obtain superior or comparable results to those reported in the state-of-the-art for the selected datasets.

Multi-Modal Physiological Data Fusion for Affect Estimation Using Deep Learning

“…The majority of the detection models are based on traditional machine learning with handcrafted features [15]- [17]. The fusion of multi-modal physiological signals [18] has shown to improve performance over ensembling multiple traditional models on a single modality [19]- [21].…”

“…In the late fusion, the outputs from the multiple models that are pretrained separately are combined to get one prediction by training a new model, majority voting or by averaging the probabilities for each class across the models then selecting the class with the highest probability. Early fusion has been adopted by Schmidt et al [17], Oh et al [14], and Bota et al [18]. In Ref.…”

“…In Ref. [18], Bota et al shows that early fusion is the most appropriate modality fusion for traditional emotion classifiers in terms of accuracy and computational complexity.…”

“…Multi-modal fusion with traditional machine learning has been explored in the literature, early fusion was shown to be the most appropriate [18]. Therefore, in this paper, we used early fusion (i.e.…”

Emotion and Stress Recognition Related Sensors and Machine Learning Technologies

“…Motivated by the limitations of emotion recognition systems in terms of lack of systematic analysis in literature regarding the selection of classifiers to use, sensor modalities, features and range of expected accuracy, and many other limitations, the work in [ 15 ] contributed to the body of work in machine learning by presenting a systematic study across five public datasets commonly used in Emotion Recognition (ER) with the objective of evaluating emotion in terms of low/high arousal and valence classification through Supervised Learning (SL), Decision Fusion (DF) and Feature Fusion (FF) techniques using multimodal physiological data, namely Electrocardiography (ECG), Electrodermal Activity (EDA), Respiration (RESP) or Blood Volume Pulse (BVP). The work considered: (i) Classification performance analysis of ER benchmarking datasets in the arousal/valence space; (ii) Summarising the ranges of the classification accuracy reported across the existing literature; (iii) Characterising the results for diverse classifiers, sensor modalities and feature set combinations for ER using accuracy and F1-score; (iv) Exploration of an extended feature set for each modality and (v) Systematic analysis of multimodal classification in DF and FF approaches.…”