Multiple Instance Learning for Emotion Recognition Using Physiological Signals

Abstract

The problem of continuous emotion recognition has been the subject of several studies. The proposed affective computing approaches employ sequential machine learning algorithms for improving the classification stage, accounting for the time ambiguity of emotional responses. Modeling and predicting the affective state over time is not a trivial problem because continuous data labeling is costly and not always feasible. This is a crucial issue in real-life applications, where data labeling is sparse and possibly captures only the most important events rather than the typical continuous subtle affective changes that occur. In this work, we introduce a framework from the machine learning literature called Multiple Instance Learning, which is able to model time intervals by capturing the presence or absence of relevant states, without the need to label the affective responses continuously (as required by standard sequential learning approaches). This choice offers a viable and natural solution for learning in a weakly supervised setting, taking into account the ambiguity of affective responses. We demonstrate the reliability of the proposed approach in a gold-standard scenario and towards real-world usage by employing an existing dataset (DEAP) and a purposely built one (Consumer). We also outline the advantages of this method with respect to standard supervised machine learning algorithms.

CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors

“…For example, Romeo et al. [ 70 ] designed an SVM-based multi-instance learning algorithm to recognize valence and arousal for each fine-grained instance and achieves of accuracy on high arousal. These kinds of methods are also widely used for fine-grained emotion recognition with different data modalities such as facial expressions [ 71 ] and vocal features [ 72 ] (e.g., pitch and loudness).…”

“…The main challenge for this kind of methods is to extract and fuse both the features inside and between instances, as the information which resides only within instances may not be enough to determine which emotion it represents. Previous works [ 70 , 73 ] use the joint loss [ 74 ] of instances and bags (instances under one video stimuli) to fuse the features inside and between instances. However, it could lead to temporal ambiguity of emotions as instances are not directly trained by their emotion labels (and instead trained by the label of bags) [ 70 ].…”

Multimodal Emotion Evaluation: A Physiological Model for Cost-Effective Emotion Classification

“…In fact, there is a plethora of opportunities where emotion recognition would be beneficial [ 7 , 8 , 9 ], from computer interaction to mental health issues (e.g., to provide emotional response monitoring during therapy). Emotions regulate body changes through activation of the central and peripheral nervous system, being this translated as behavioral (facial expression, speech, posture) and/or physiological responses (electroencephalogram (EEG), electrocardiogram (ECG), electromyogram (EMG)) [ 6 , 10 ]. For instance, in the presence of emotional triggers, our heart activity changes, our facial expression is different, and our muscles compress [ 11 ].…”

“…For instance, in the presence of emotional triggers, our heart activity changes, our facial expression is different, and our muscles compress [ 11 ]. So, emotional responses may be described by the quantification of body information [ 10 , 11 ].…”

“…The literature usually presents emotion identification systems based on one signal [ 6 , 9 , 10 , 12 , 13 ]. Notwithstanding, it is known that some signals present a premature response to emotion in comparison with others [ 11 ].…”