Emotion Classification from Speech and Text in Videos Using a Multimodal Approach

Caschera, Maria Chiara; Grifoni, Patrizia; Ferri, Fernando

doi:10.3390/mti6040028

Cited by 17 publications

(2 citation statements)

References 105 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The most prominent emotion linked to eyebrow movement is that of surprise. This association has often been noted [45][46][47][48] and is, in a way, archetypical from an evolutionary point of view. Eyebrows are raised as a result of opening one's eyes wide, as a symptom of "attentional activity" [20,[48][49][50].…”

Section: Facial Gestures and Audiovisual Prosodymentioning

confidence: 82%

The Temporal Alignment of Speech-Accompanying Eyebrow Movement and Voice Pitch: A Study Based on Late Night Show Interviews

Gast

2023

Behavioral Sciences

View full text Add to dashboard Cite

Previous research has shown that eyebrow movement during speech exhibits a systematic relationship with intonation: brow raises tend to be aligned with pitch accents, typically preceding them. The present study approaches the question of temporal alignment between brow movement and intonation from a new angle. The study makes use of footage from the Late Night Show with David Letterman, processed with 3D facial landmark detection. Pitch is modeled as a sinusoidal function whose parameters are correlated with the maximum height of the eyebrows in a brow raise. The results confirm some previous findings on audiovisual prosody but lead to new insights as well. First, the shape of the pitch signal in a region of approx. 630 ms before the brow raise is not random and tends to display a specific shape. Second, while being less informative than the post-peak pitch, the pitch signal in the pre-peak region also exhibits correlations with the magnitude of the associated brow raises. Both of these results point to early preparatory action in the speech signal, calling into question the visual-precedes-acoustic assumption. The results are interpreted as supporting a unified view of gesture/speech co-production that regards both signals as manifestations of a single communicative act.

show abstract

Section: Facial Gestures and Audiovisual Prosodymentioning

confidence: 82%

The Temporal Alignment of Speech-Accompanying Eyebrow Movement and Voice Pitch: A Study Based on Late Night Show Interviews

Gast

2023

Behavioral Sciences

View full text Add to dashboard Cite

show abstract

“…Rao et al (2021) have employed speech-based LSTM features, k-nearest neighbors (kNNs), Bayesian networks, hidden Markov models (HMMs), and artificial neural networks (ANN) based features from facial expressions for acoustic features (Gaussian mixture model, Mel frequency cepstral coefficients) using RAVDESS (Ryerson audio-visual database of emotional speech and song) audio dataset. Emotion classification from speech and text in videos using a multimodal approach has been performed in Caschera et al (2022) where an automatic extraction of emotional information from a variety of data provided by different interaction modalities and from different domains has been demonstrated. A survey on multimodal video sentiment analysis using deep learning approaches has been reported in Abdu et al (2021) where multimodal sentiment analysis systems with the Multimodal Multi-Utterance based architecture have been discussed.…”

Section: Multimodal Sentiment Analysismentioning

confidence: 99%

A multimodal sentiment analysis system for recognizing person aggressiveness in pain based on textual and visual information

Ghosh

Dhara²,

Pero³

et al. 2023

J Ambient Intell Human Comput

View full text Add to dashboard Cite

This article proposes a multimodal sentiment analysis system for recognizing a person’s aggressiveness in pain. The implementation has been divided into five components. The first three steps are related to a text-based sentiment analysis system to perform classification tasks such as predicting the classes into non-aggressive, covertly aggressive, and overtly aggressive classes. The remaining two components are related to an image-based sentiment analysis system. A deep learning-based approach has been employed to do feature learning and predict the three types of pain classes. An aggression dataset for the text-based system and the UNBC-McMaster database for an image-based system has been employed, respectively. Experimental results have been compared with the state-of-the-art methods, showing the superiority of the proposed approach. Finally, the scores due to text-based and image-based sentiment analysis systems are fused to obtain the performance for the proposed multimodal sentiment analysis system.

show abstract

Multimodal emotion recognition: A comprehensive review, trends, and challenges

Ramaswamy,

Palaniswamy

2024

WIREs Data Min & Knowl

View full text Add to dashboard Cite

Automatic emotion recognition is a burgeoning field of research and has its roots in psychology and cognitive science. This article comprehensively reviews multimodal emotion recognition, covering various aspects such as emotion theories, discrete and dimensional models, emotional response systems, datasets, and current trends. This article reviewed 179 multimodal emotion recognition literature papers from 2017 to 2023 to reflect on the current trends in multimodal affective computing. This article covers various modalities used in emotion recognition based on the emotional response system under four categories: subjective experience comprising text and self‐report; peripheral physiology comprising electrodermal, cardiovascular, facial muscle, and respiration activity; central physiology comprising EEG, neuroimaging, and EOG; behavior comprising facial, vocal, whole‐body behavior, and observer ratings. This review summarizes the measures and behavior of each modality under various emotional states. This article provides an extensive list of multimodal datasets and their unique characteristics. The recent advances in multimodal emotion recognition are grouped based on the research focus areas such as emotion elicitation strategy, data collection and handling, the impact of culture and modality on multimodal emotion recognition systems, feature extraction, feature selection, alignment of signals across the modalities, and fusion strategies. The recent multimodal fusion strategies are detailed in this article, as extracting shared representations of different modalities, removing redundant features from different modalities, and learning critical features from each modality are crucial for multimodal emotion recognition. This article summarizes the strengths and weaknesses of multimodal emotion recognition based on the review outcome, along with challenges and future work in multimodal emotion recognition. This article aims to serve as a lucid introduction, covering all aspects of multimodal emotion recognition for novices.This article is categorized under: Fundamental Concepts of Data and Knowledge > Human Centricity and User Interaction Technologies > Cognitive Computing Technologies > Artificial Intelligence

show abstract

Emotion Classification from Speech and Text in Videos Using a Multimodal Approach

Cited by 17 publications

References 105 publications

The Temporal Alignment of Speech-Accompanying Eyebrow Movement and Voice Pitch: A Study Based on Late Night Show Interviews

The Temporal Alignment of Speech-Accompanying Eyebrow Movement and Voice Pitch: A Study Based on Late Night Show Interviews

A multimodal sentiment analysis system for recognizing person aggressiveness in pain based on textual and visual information

Multimodal emotion recognition: A comprehensive review, trends, and challenges

Contact Info

Product

Resources

About