Weakly-Supervised Learning for Fine-Grained Emotion Recognition Using Physiological Signals

Zhang, Tianyi; Ali, Abdallah El; Wang, Chen; Hanjalic, Alan; César, Pablo

doi:10.1109/taffc.2022.3158234

Cited by 10 publications

(8 citation statements)

References 110 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The results in Figure 3 suggest that depressed patients may only display affective state associated with depressive level for a small fraction of the time. In the field of emotion recognition using physiological signals, a weakly supervised learning work [58] has also shown that subjects' emotion labels while watching videos are represented only by the most salient or recent emotion. In addition, by analyzing the attention weights between the two groups, we found that depression subjects and healthy subjects have significant differences in their responses to negative video stimulation.…”

Section: Discussionmentioning

confidence: 99%

Dual-Stream Multiple Instance Learning for Depression Detection With Facial Expression Videos

Shangguan

Liu

Li³

et al. 2023

IEEE Trans. Neural Syst. Rehabil. Eng.

View full text Add to dashboard Cite

Depression is a common mental illness which has brought great harm to the individuals. With recent evidence that many objective physiological signals are associated with depression, automated detection of depression is urgent and important for the growing concern of mental illness. We investigate the problem of classifying depression by facial expressions, which may aid in online diagnosis and rehabilitation engineering of depression. In this work, We propose a weakly supervised learning approach employing multiple instance learning (MIL) on 150 videos data from 75 depressed and 75 healthy subjects.In addition, we present a novel MIL dual-stream aggregator that considers both the instance-level and the bag-level in order to emphasize the information with symptoms. Specifically, our method named ADDMIL uses max-pooling at the instance level to capture symptom information and further integrates the contribution of each instance at the bag level using attention weights. Our method achieves 74.7% accuracy and 74.5% recall on the collected dataset, which not only improves 10.1% accuracy and 9.8% recall over the baseline but also exceeds the best accuracy result of MIL-based method by 2.1%. Our work achieves results that are comparable to the state-of-the-art methods and demonstrates that multiple instance learning has great potential for depression classification. We present for the first time a weakly supervised learning approach in the detection of depression through raw facial expressions, which may provide a new framework for other psychiatric disorders detection methods.

show abstract

Section: Discussionmentioning

confidence: 99%

Dual-Stream Multiple Instance Learning for Depression Detection With Facial Expression Videos

Shangguan

Liu

Li³

et al. 2023

IEEE Trans. Neural Syst. Rehabil. Eng.

View full text Add to dashboard Cite

show abstract

“…Tests achieved an averaged accuracy of 76.04% (arousal), 76.62% (valence), and 57.62% (four quadrant with neutral). Additionally, Zhang et al [24] proposed a method based on deep multiple instance learning, to recognize emotions at a finer granularity level when trained with post-stimuli labels, where instances are weakly-supervised in training stage. In Elalamy et al [25] work, recurrence plots were used to obtain 2D representations of physiological activity, to get less subject dependent and better suited representation for non-stationary signals such as EDA, in conjunction with ECG and PPG.…”

Section: A Peripheral Signals-based Methodsmentioning

confidence: 99%

LSTM-Modeling of Emotion Recognition Using Peripheral Physiological Signals in Naturalistic Conversations

Zitouni

Park

Lee

et al. 2023

IEEE J. Biomed. Health Inform.

View full text Add to dashboard Cite

The automated recognition of human emotions plays an important role in developing machines with emotional intelligence. Major research efforts are dedicated to the development of emotion recognition methods. However, most of the affective computing models are based on images, audio, videos and brain signals. Literature lacks works that focus on utilizing only peripheral signals for emotion recognition (ER), which can be ideally implemented in daily life settings. Therefore, this paper present a framework for ER on the arousal and valence space, based on using multi-modal peripheral signals. The data used in this work were collected during a debate between two people using wearable devices. The emotions of the participants were rated by multiple raters and converted into classes in correspondence to the arousal and valence space. The use of a dynamic threshold for ratings conversion was investigated. An ER model is proposed that uses a Long Short-Term Memory (LSTM)-based architecture for classification. The model uses heart rate (HR), temperature (T), and electrodermal activity (EDA) signals as its inputs with emotional cues. Additionally, a post-processing prediction mechanism is introduced to enhance the recognition performance. The model is implemented to study the use of individual and different combinations of the peripheral signals, as well as utilizing annotations from different ratings. Additionally, it is employed for classification of valence and arousal in an independent and combined fashion, under subject dependent and independent scenarios. The experimental results have justified the efficient performance of the proposed framework, achieving classification accuracy >96% and >93% for the independent and

show abstract

“…Despite the continuous nature of the emotion labels in the RECOLA and CASE datasets, some works utilize these to only predict discrete categories. Zhang et al [42] for instance, used deep multiple instance learning to classify labels in the CASE dataset into the low, neutral and high categories. Here, classification was based on features extracted from skin conductance, blood volume pulse (BVP), skin temperature and heart rate.…”

Section: A Related Work and Limitations -Machine Learning Methodsmentioning

confidence: 99%

A Bayesian Filtering Approach for Tracking Sympathetic Arousal and Cortisol-Related Energy From Marked Point Process and Continuous-Valued Observations

Wickramasuriya,

Khazaei,

Kiani

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Multiple state variables governed by internal processes within the human body remain unobserved. On a number of occasions, these states are linked to point process bioelectric and biochemical observations coupled together with continuous-valued variables. These observations provide a means to estimate the latent states of interest. We develop a state-space method to estimate unobserved sympathetic arousal and energy production states from skin conductance and cortisol data respectively, comprising of a marked point process and a continuous-valued observation. The method involves Bayesian filtering applied within an expectation-maximization (EM) framework for state estimation and model parameter recovery. Results are evaluated on both simulated and experimental data. On experimental skin conductance data, high arousal levels are generally detected during cognitive stress periods and lower values are detected during relaxation. Results are also in conformity with general physiology for cortisol data. On separate experimental data, skin conductance-based estimates are validated/cross-checked with functional Near Infrared Spectroscopy features. Estimation is also performed on heart rate and skin conductance data to illustrate the method's wider applicability. We also compare the method with earlier approaches. We show how it outperforms a previous method for cortisol-based energy estimation, and its superiority to earlier methods for estimating sympathetic arousal. The EM approach is thus able to estimate latent physiological states within the body from point process bioelectric and biochemical phenomena. The method could be applied in wearable monitoring and automated closed-loop therapy delivery for patients diagnosed with certain types of neuropsychiatric or hormone disorders.INDEX TERMS Biomedical signal processing, state-space methods, estimation, emotion recognition, Kalman filters I. INTRODUCTIONS EVERAL internal state variables maintained by physiological processes within the human body remain largely unobserved. Nevertheless, they are often tied to more easilymeasured changes in biolelectric and biochemical phenomena that can be used for their estimation. For instance, energy

show abstract

Weakly-Supervised Learning for Fine-Grained Emotion Recognition Using Physiological Signals

Cited by 10 publications

References 110 publications

Dual-Stream Multiple Instance Learning for Depression Detection With Facial Expression Videos

Dual-Stream Multiple Instance Learning for Depression Detection With Facial Expression Videos

LSTM-Modeling of Emotion Recognition Using Peripheral Physiological Signals in Naturalistic Conversations

A Bayesian Filtering Approach for Tracking Sympathetic Arousal and Cortisol-Related Energy From Marked Point Process and Continuous-Valued Observations

Contact Info

Product

Resources

About