Building a three-level multimodal emotion recognition framework

Abstract

Multimodal emotion detection has been one of the main lines of research in the field of Affective Computing (AC) in recent years. Multimodal detectors aggregate information coming from different channels or modalities to determine what emotion users are expressing with a higher degree of accuracy. However, despite the benefits offered by this kind of detectors, their presence in real implementations is still scarce for various reasons. In this paper, we propose a technology-agnostic framework, HERA, to facilitate the creation of multimodal emotion detectors, offering a tool characterized by its modularity and the interface-based programming approach adopted in its development. HERA (Heterogeneous Emotional Results Aggregator) offers an architecture to integrate different emotion detection services and aggregate their heterogeneous results to produce a final result using a common format. This proposal constitutes a step forward in the development of multimodal detectors, providing an architecture to manage different detectors and fuse the results produced by them in a sensible way. We assessed the validity of the proposal by testing the system with several developers with no previous knowledge about affective technology and emotion detection. The assessment was performed applying the Computer System Usability Questionnaire and the Twelve Cognitive Dimensions Questionnaire, used by The Visual Studio Usability group at Microsoft, obtaining positive results and important feedback for future versions of the system.

Multimodal Sentiment Analysis Method Based on Hierarchical Adaptive Feature Fusion Network

“…Early MSA often focused on single modality information such as sound, text, visual, and biological signals. However, using a single modality for MSA users often did not accurately analyze their sentiments (Salhi et al, 2021;Mohammed et al, 2022;Garcia-Garcia, 2023). Because the same text may express opposite meanings in different contexts, it is difficult to accurately predict users' sentiments based solely on one modality.…”

An Assessment of In-the-Wild Datasets for Multimodal Emotion Recognition

“…However, despite the progress of ER using Deep Learning techniques that is shown in a large number of studies, most of them use datasets built in laboratory environments, such as IEMOCAP [20,23], AMIGOS [24], RECOLA [28], DEAP [21], SEED, SEED-IV, SEED-V, DEAP and DREAMER [25], etc. The fundamental problem with any recognition system is the lack of data or the training with real data which may affect its generalization to examples that have not been seen during the training process [21,35]. Furthermore, the datasets used for training of ER models have been designed in controlled laboratory environments and are significantly different from what happens in real conditions in terms of brightness, noise level, etc.…”