Decision-Level Fusion Method for Emotion Recognition using Multimodal Emotion Recognition Information

Song, Kyu-Seob; Nho, Young-Hoon; Seo, Jongwon; Kwon, Dong‐Soo

doi:10.1109/urai.2018.8441795

Cited by 20 publications

(6 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data fusion is a critical step involved in multimodal emotion recognition for producing the estimation. The literature about emotional data fusion involves three data fusion techniques, which are early fusion (feature fusion) [42,43], late fusion (decision fusion) [44,45,46] and hybrid approaches [17,47,48].…”

Section: Background and Literature Review On Multimodal Emotion Rmentioning

confidence: 99%

See 1 more Smart Citation

Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion

2020

View full text Add to dashboard Cite

Multimodal emotion recognition has gained traction in affective computing research community to overcome the limitations posed by the processing a single form of data and to increase recognition robustness. In this study, a novel emotion recognition system is introduced, which is based on multiple modalities including facial expressions, galvanic skin response (GSR) and electroencephalogram (EEG). This method follows a hybrid fusion strategy and yields a maximum one-subject-out accuracy of 81.2% and a mean accuracy of 74.2% on our bespoke multimodal emotion dataset (LUMED-2) for 3 emotion classes: sad, neutral and happy. Similarly, our approach yields a maximum one-subject-out accuracy of 91.5% and a mean accuracy of 53.8% on the Database for Emotion Analysis using Physiological Signals (DEAP) for varying numbers of emotion classes, 4 in average, including angry, disgust, afraid, happy, neutral, sad and surprised. The presented model is particularly useful in determining the correct emotional state in the case of natural deceptive facial expressions. In terms of emotion recognition accuracy, this study is superior to, or on par with, the reference subject-independent multimodal emotion recognition studies introduced in the literature.

show abstract

Section: Background and Literature Review On Multimodal Emotion Rmentioning

confidence: 99%

“…These outputs can be used separately or jointly through machine learning methods. Some studies have implemented decision level fusion as reported in [44,45,49,54,55].…”

Section: Background and Literature Review On Multimodal Emotion Rmentioning

confidence: 99%

Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion

2020

View full text Add to dashboard Cite

show abstract

“…By integrating faces, gestures, and voice at the decision level, the system could better understand user's intentions. Song et al [35] used K-NN as a classifier in the decision-making layer for the facial expression and speech emotion recognition results, which improved the recognition rate of existing emotion recognizers on social robots. In summary, the multimodal fusion method is used to eliminate the ambiguity of another single mode through a single mode.…”

Section: Related Workmentioning

confidence: 99%

Research on Multimodal Perceptual Navigational Virtual and Real Fusion Intelligent Experiment Equipment and Algorithm

et al. 2020

View full text Add to dashboard Cite

Virtual experiment is an important field of human-computer interaction. With more and more virtual laboratories emerging, we found that problems regarding virtual experiments are rising. Such problems can be listed as follows: First, human-computer interaction has lower efficiency during the process of virtual experiment, which means the computer cannot understand the user's intention thus leading to incorrect operation. Second, there are less detections for false behavior during experiments. Third, the virtual laboratory's sense of operation and realism is not strong. In order to solve the above problems, the multimodal sensing navigation virtual and real fusion laboratory (MSNVRFL) was designed and implemented in this paper. We design a new set of experimental equipment with the function of cognition and study a multimodal fusion model and algorithm for chemical experiments, which are both finally verified and applied in MSNVRFL. By using multimodal fusion perception algorithm, the user's true intentions can be understood and the human-computer interaction efficiency can be improved. By carrying out a virtual experiment with the mold of virtual and real fusion, problems like resources wasting and dangers happened during experiment can be avoided, user's sense of operation and realism can be improved. In addition, teaching navigation and wrong operation behavior reminders are provided for users. The experimental result shows that our method can improve the efficiency of human-computer interaction, reduce the user's cognitive load, strengthen the user's sense of reality and operation and stimulate students' interest in learning. INDEX TERMS Multimodal fusion, virtual experiments, intelligent teaching, human-computer interaction.

show abstract

“…Emotion recognition from features such as facial expressions, gestures, and walks has been addressed in the literature surveyed in [1,2,4,29]. Multimodal and context-aware affect recognition models are also available for that purpose as well [19][20][21]33].…”

Section: Multimodal Behavior Analysis In Social Roboticsmentioning

confidence: 99%

SeekNet: Improved Human Instance Segmentation and Tracking via Reinforcement Learning Based Optimized Robot Relocation

Narayanan¹,

Manoghar²,

Prashanth³

et al. 2020

Preprint

View full text Add to dashboard Cite

Amodal recognition is the ability of the system to detect occluded objects. Most state-of-the-art Visual Recognition systems lack the ability to perform amodal recognition. Few studies have achieved amodal recognition through passive prediction or embodied recognition approaches. However, these approaches suffer from challenges in real-world applications, such as dynamic objects. We propose SeekNet , an improved optimization method for amodal recognition through embodied visual recognition. Additionally, we implement SeekNet for social robots, where there are multiple interactions with crowded humans. Hence, we focus on occluded human detection & tracking and showcase the superiority of our algorithm over other baselines. We also experiment with SeekNet to improve the confidence of COVID-19 symptoms pre-screening algorithms using our efficient embodied recognition system.

show abstract

Decision-Level Fusion Method for Emotion Recognition using Multimodal Emotion Recognition Information

Cited by 20 publications

References 11 publications

Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion

Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion

Research on Multimodal Perceptual Navigational Virtual and Real Fusion Intelligent Experiment Equipment and Algorithm

SeekNet: Improved Human Instance Segmentation and Tracking via Reinforcement Learning Based Optimized Robot Relocation

Contact Info

Product

Resources

About