Multimodal Emotion Recognition

Haq, Sanaul; Jackson, Philip J.

doi:10.4018/9781615209194.ch017

Cited by 3 publications

(4 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…LDA plays an essential role in improving the performance of the emotion recognition system. 35,39 On top of that, it is worth noting that the results obtained using ECG image data also showed positive potential and could be considered as an input modality for the ERS. The features extracted by KAZE provided 63% accuracy for valence, which is better than the original work in Ref.…”

Section: Discussionmentioning

confidence: 89%

Evaluation of electrocardiogram: numerical vs. image data for emotion recognition system

et al. 2021

View full text Add to dashboard Cite

Background: The electrocardiogram (ECG) is a physiological signal used to diagnose and monitor cardiovascular disease, usually using ECG wave images. Numerous studies have proven that ECG can be used to detect human emotions using numerical data; however, ECG is typically captured as a wave image rather than as a numerical data. There is still no consensus on the effect of the ECG input format (either as an image or a numerical value) on the accuracy of the emotion recognition system (ERS). The ERS using ECG images is still inadequately studied. Therefore, this study compared ERS performance using ECG image and ECG numerical data to determine the effect of the ECG input format on the ERS. Methods: This study employed the DREAMER dataset, which contains 23 ECG recordings obtained during audio-visual emotional elicitation. Numerical data was converted to ECG images for the comparison. Numerous approaches were used to obtain ECG features. The Augsburg BioSignal Toolbox (AUBT) and the Toolbox for Emotional feature extraction from Physiological signals (TEAP) extracted features from numerical data. Meanwhile, features were extracted from image data using Oriented FAST and rotated BRIEF (ORB), Scale Invariant Feature Transform (SIFT), KAZE, Accelerated-KAZE (AKAZE), Binary Robust Invariant Scalable Keypoints (BRISK), and Histogram of Oriented Gradients (HOG). Dimension reduction was accomplished using linear discriminant analysis (LDA), and valence and arousal were classified using the Support Vector Machine (SVM). Results: The experimental results indicated that numerical data achieved arousal and valence accuracy of 69% and 79%, respectively, which was greater than those of image data. For ECG images, the highest accuracy for arousal was 58% percent; meanwhile, the valence was 63%. Conclusions: The finding showed that numerical data provided better accuracy for ERS. However, ECG image data which shows positive potential and can be considered as an input modality for the ERS.

show abstract

Section: Discussionmentioning

confidence: 89%

Evaluation of electrocardiogram: numerical vs. image data for emotion recognition system

et al. 2021

View full text Add to dashboard Cite

show abstract

“…This is seen in the Aff-Wild ( Zafeiriou et al, 2017 ) and AFEW-VA ( Kossaifi et al, 2017 ) datasets where the audio samples are affected by background music or noise. On the other hand, datasets like RAVDESS ( Livingstone and Russo, 2018 ) and SAVEE ( Haq and Jackson, 2010 ) provide clean audio and video samples but use categorical labelling.…”

Section: Training and Evaluationmentioning

confidence: 99%

Affect-Driven Learning of Robot Behaviour for Collaborative Human-Robot Interactions

et al. 2022

View full text Add to dashboard Cite

Collaborative interactions require social robots to share the users’ perspective on the interactions and adapt to the dynamics of their affective behaviour. Yet, current approaches for affective behaviour generation in robots focus on instantaneous perception to generate a one-to-one mapping between observed human expressions and static robot actions. In this paper, we propose a novel framework for affect-driven behaviour generation in social robots. The framework consists of (i) a hybrid neural model for evaluating facial expressions and speech of the users, forming intrinsic affective representations in the robot, (ii) an Affective Core, that employs self-organising neural models to embed behavioural traits like patience and emotional actuation that modulate the robot’s affective appraisal, and (iii) a Reinforcement Learning model that uses the robot’s appraisal to learn interaction behaviour. We investigate the effect of modelling different affective core dispositions on the affective appraisal and use this affective appraisal as the motivation to generate robot behaviours. For evaluation, we conduct a user study (n = 31) where the NICO robot acts as a proposer in the Ultimatum Game. The effect of the robot’s affective core on its negotiation strategy is witnessed by participants, who rank a patient robot with high emotional actuation higher on persistence, while an impatient robot with low emotional actuation is rated higher on its generosity and altruistic behaviour.

show abstract

“…The emotion recognition becomes essential and useful approach in Human Computer Interaction and Human Robot Interaction applications because emotion can indicate human feeling and requirement [1]. The application can recognize user emotion to provide service.…”

Section: Emotion Recognitionmentioning

confidence: 99%

“…The positive emotion (pleasant) is happiness, pleasure and satisfaction. The negative emotion (unpleasant) is sadness, anger, anxiety and fear [1]. People with good emotional health can control themselves and can address their negative emotions.…”

Section: Introductionmentioning

confidence: 99%

Healthcare System Focusing on Emotional Aspect using Augmented Reality

Tivatansakul

Ohkura

2014

Transactions of Japan Society of Kansei Engineering

View full text Add to dashboard Cite

Even though much recent research has proposed systems that provide assistance and services to people, these systems emphasize the support of the physical aspects at the expense of emotional aspects. However, emotional health is as important as physical health. Moreover, negative emotional health can lead to social or mental health problems. People with good emotional health can control themselves and can address their negative emotions. However, people with bad emotional health feel difficult to control and decrease their negative emotions. Therefore the negative emotional health can lead to social and mental health problems. To cope with negative emotional health in their daily life, we proposed a new healthcare system that focuses on emotional aspects. The system integrates augmented reality to display virtual objects in real environments and Kinect, which allows users to freely interact with them. We also employ biological sensors to measure and detect user emotions, and provide three services based on expected emotions: relaxation, amusement and excitement services. We designed and implemented a small prototype for our relaxation service called: a breathing control application. This application applies deep breathing techniques of stress management to supports users when they experience stress from society or work as well as other negative emotions. This application displays a virtual music box to assist them breathe deeply because virtual objects and music might increase user relaxation and decrease their stress. We also evaluated this application by performing the experiment. The stress and relaxation of the participants are measured using salivary amylase test. As the result, the application can decrease their stress.

show abstract

Multimodal Emotion Recognition

Cited by 3 publications

References 9 publications

Evaluation of electrocardiogram: numerical vs. image data for emotion recognition system

Evaluation of electrocardiogram: numerical vs. image data for emotion recognition system

Affect-Driven Learning of Robot Behaviour for Collaborative Human-Robot Interactions

Healthcare System Focusing on Emotional Aspect using Augmented Reality

Contact Info

Product

Resources

About