A Survey on Databases for Multimodal Emotion Recognition and an Introduction to the VIRI (Visible and InfraRed Image) Database

Abstract

Multimodal human–computer interaction (HCI) systems pledge a more human–human-like interaction between machines and humans. Their prowess in emanating an unambiguous information exchange between the two makes these systems more reliable, efficient, less error prone, and capable of solving complex tasks. Emotion recognition is a realm of HCI that follows multimodality to achieve accurate and natural results. The prodigious use of affective identification in e-learning, marketing, security, health sciences, etc., has increased demand for high-precision emotion recognition systems. Machine learning (ML) is getting its feet wet to ameliorate the process by tweaking the architectures or wielding high-quality databases (DB). This paper presents a survey of such DBs that are being used to develop multimodal emotion recognition (MER) systems. The survey illustrates the DBs that contain multi-channel data, such as facial expressions, speech, physiological signals, body movements, gestures, and lexical features. Few unimodal DBs are also discussed that work in conjunction with other DBs for affect recognition. Further, VIRI, a new DB of visible and infrared (IR) images of subjects expressing five emotions in an uncontrolled, real-world environment, is presented. A rationale for the superiority of the presented corpus over the existing ones is instituted.

Emotion Intensity Detection in Online Media: An Attention Mechanism Based Multimodal Deep Learning Approach

“…There are several main methods for sentiment analysis of multimodal data [13]: 1) Fusion feature methods: the features of different modalities are fused, and then sentiment analysis is performed using traditional machine learning algorithms or deep learning models. Common fusion feature methods include feature-level fusion and decision-level fusion [13]. 2) Multimodal feature learning methods: mapping features of different modalities into a shared feature space by learning the correlation between multimodal data.…”

“…Multimodal data usually contains different types of information, such as text, images, audio, etc. The attention mechanism can help the model automatically learn the level of attention between different modalities and determine how much each modality contributes to the sentiment analysis [13]. This can better utilize the information of different modalities and improve the accuracy and performance of sentiment analysis.…”

The Multimodal Emotion Information Analysis of E-Commerce Online Pricing in Electronic Word of Mouth

“…The video data is divided into speech and image data, and the emotional features of the two are extracted respectively to obtain the expression recognition model of a single feature. Then, the two models are fused to obtain a multimodal integrated expression recognition model (Siddiqui et al, 2022). Figure 1 shows the specific process.…”

CG-MER: a card game-based multimodal dataset for emotion recognition

“…These unimodal datasets have significantly contributed to advancing emotion recognition models within their respective modalities. However, it is important to acknowledge that unimodal datasets may not cover the complete spectrum of emotions expressed through multiple modalities [10]. To overcome this limitation, using multimodal datasets has become crucial, as they provide a comprehensive understanding of emotions.…”