Multimodal Affect Recognition in the Context of Human-Computer Interaction for Companion-Systems

Schwenker, Friedhelm; Böck, Ronald; Schels, Martin; Meudt, Sascha; Siegert, Ingo; Glodek, Michael; Kächele, Markus; Schmidt-Wack, Miriam; Thiam, Patrick; Wendemuth, Andreas; Krell, Gerald

doi:10.1007/978-3-319-43665-4_19

Cited by 3 publications

(3 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For an excellent overview of the application of deep learning and as well as shallow learning approaches to FER, the reader is directed to [ 15 ] and the references there in. Additionally, the reader can refer to this Chapter on Multimodal Affect Recognition in the Context of Human-Computer Interaction [ 16 ].…”

Section: Introductionmentioning

confidence: 99%

FusionSense: Emotion Classification Using Feature Fusion of Multimodal Data and Deep Learning in a Brain-Inspired Spiking Neural Network

Tan

Ceballos

Kasabov

et al. 2020

Sensors

View full text Add to dashboard Cite

Using multimodal signals to solve the problem of emotion recognition is one of the emerging trends in affective computing. Several studies have utilized state of the art deep learning methods and combined physiological signals, such as the electrocardiogram (EEG), electroencephalogram (ECG), skin temperature, along with facial expressions, voice, posture to name a few, in order to classify emotions. Spiking neural networks (SNNs) represent the third generation of neural networks and employ biologically plausible models of neurons. SNNs have been shown to handle Spatio-temporal data, which is essentially the nature of the data encountered in emotion recognition problem, in an efficient manner. In this work, for the first time, we propose the application of SNNs in order to solve the emotion recognition problem with the multimodal dataset. Specifically, we use the NeuCube framework, which employs an evolving SNN architecture to classify emotional valence and evaluate the performance of our approach on the MAHNOB-HCI dataset. The multimodal data used in our work consists of facial expressions along with physiological signals such as ECG, skin temperature, skin conductance, respiration signal, mouth length, and pupil size. We perform classification under the Leave-One-Subject-Out (LOSO) cross-validation mode. Our results show that the proposed approach achieves an accuracy of 73.15% for classifying binary valence when applying feature-level fusion, which is comparable to other deep learning methods. We achieve this accuracy even without using EEG, which other deep learning methods have relied on to achieve this level of accuracy. In conclusion, we have demonstrated that the SNN can be successfully used for solving the emotion recognition problem with multimodal data and also provide directions for future research utilizing SNN for Affective computing. In addition to the good accuracy, the SNN recognition system is requires incrementally trainable on new data in an adaptive way. It only one pass training, which makes it suitable for practical and on-line applications. These features are not manifested in other methods for this problem.

show abstract

Section: Introductionmentioning

confidence: 99%

FusionSense: Emotion Classification Using Feature Fusion of Multimodal Data and Deep Learning in a Brain-Inspired Spiking Neural Network

Tan

Ceballos

Kasabov

et al. 2020

Sensors

View full text Add to dashboard Cite

show abstract

“…This ever-increasing technologisation brings numerous challenges pertaining to effective and continuously engaging user-interaction. Addressing these challenges effectively is the goal of Affective Computing (AC), which aims to develop systems that can recognise and process human emotions [26], such that they continuously adapt to the user's needs [31]. To safety [38], and a study on the use of AC methods for continuous pain intensity assessment [21].…”

Section: Introductionmentioning

confidence: 99%

“…This is still largely an unsolved problem, but a commonly followed approach in laboratory settings involves eliciting emotional response from humans using stimuli like pictures [25], videos [33,40], music [25], etc., while simultaneously acquiring modalities and annotations pertaining to the emotional experience. These annotations are usually provided in form of either discrete emotion categories [31,40] (e.g., fear, joy, etc.) or in terms of Valence and Arousal (V-A) values as per the continuous 2-dimensional Circumplex model of Affect [30].…”

Section: Introductionmentioning

confidence: 99%

A functional data analysis approach for continuous 2-D emotion annotations

Sharma

Wagner

Castellini

et al. 2019

WEB

Self Cite

View full text Add to dashboard Cite

The standard paradigm in Affective Computing involves acquiring one/several markers (e.g., physiological signals) of emotions and training models on these to predict emotions. However, due to the internal nature of emotions, labelling/annotation of emotional experience is done manually by humans using specially developed annotation tools. To effectively exploit the resulting subjective annotations for developing affective systems, their quality needs to be assessed. This entails, (i) evaluating the variations in annotations, across different subjects and emotional stimuli, to detect spurious/unexpected patterns; and (ii) developing strategies to effectively combine these subjective annotations into a ground truth annotation. This article builds on our previous work by presenting a novel Functional Data Analysis based approach to assess the quality of annotations. Specifically, the bivariate annotation time-series are transformed into functions, such that each resulting functional annotation then becomes a sample element for analysis like Multivariate Functional Principal Component Analysis (MFPCA) that evaluate variation across all annotations. The resulting scores from MFPCA provide interesting insights into annotation patterns and facilitate the use of multivariate statistical techniques to address both (i) and (ii). Given the presented efficacy of these methods, we believe they offer an exciting new approach to assessing the quality of annotations.

show abstract

Deep Learning Algorithms for Emotion Recognition on Low Power Single Board Computers

Srinivasan

Meudt

Schwenker

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Multimodal Affect Recognition in the Context of Human-Computer Interaction for Companion-Systems

Cited by 3 publications

References 40 publications

FusionSense: Emotion Classification Using Feature Fusion of Multimodal Data and Deep Learning in a Brain-Inspired Spiking Neural Network

FusionSense: Emotion Classification Using Feature Fusion of Multimodal Data and Deep Learning in a Brain-Inspired Spiking Neural Network

A functional data analysis approach for continuous 2-D emotion annotations

Deep Learning Algorithms for Emotion Recognition on Low Power Single Board Computers

Contact Info

Product

Resources

About