Recognizing Emotions evoked by Movies using Multitask Learning

Hayat, Hassan; Ventura, Carles; Lapedriza, Àgata

doi:10.1109/acii52823.2021.9597464

Cited by 2 publications

(4 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We ensure that each movie should be in the test set. In comparison with a previous study [ 7 ], the audio modality has been also added to the backbone architecture besides the visual and text modalities. The Multi-Task approach showed better generalization for each annotator.…”

Section: Resultsmentioning

confidence: 99%

“…A deeper analysis of the results on the COGNIMUSE dataset can be found in a previous work [ 7 ]. This analysis includes the results for each partition considered in the cross-validation evaluation as well as an ablation study of the different modalities.…”

Section: Resultsmentioning

confidence: 99%

“…This work is based on a previous study on modeling the emotions evoked by movies [ 7 ]. In [ 7 ], the authors proposed a specific bi-modal emotion modeling technique that jointly modeled the emotions experienced by different people while watching a movie in a Multi-Task manner. This Multi-Task approach showed better results than modeling each person separately with a Single-Task approach [ 7 ].…”

Section: Introductionmentioning

confidence: 99%

“…In [ 7 ], the authors proposed a specific bi-modal emotion modeling technique that jointly modeled the emotions experienced by different people while watching a movie in a Multi-Task manner. This Multi-Task approach showed better results than modeling each person separately with a Single-Task approach [ 7 ]. While this previous study focused on the emotions evoked by movies combining visual and text modality, this work extends the modeling approach to a more generic formulation, and presents further experiments in more affect-related datasets.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Modeling Subjective Affect Annotations with Multi-Task Learning

Hayat

Ventura

Lapedriza

2022

Sensors

Self Cite

View full text Add to dashboard Cite

In supervised learning, the generalization capabilities of trained models are based on the available annotations. Usually, multiple annotators are asked to annotate the dataset samples and, then, the common practice is to aggregate the different annotations by computing average scores or majority voting, and train and test models on these aggregated annotations. However, this practice is not suitable for all types of problems, especially when the subjective information of each annotator matters for the task modeling. For example, emotions experienced while watching a video or evoked by other sources of content, such as news headlines, are subjective: different individuals might perceive or experience different emotions. The aggregated annotations in emotion modeling may lose the subjective information and actually represent an annotation bias. In this paper, we highlight the weaknesses of models that are trained on aggregated annotations for modeling tasks related to affect. More concretely, we compare two generic Deep Learning architectures: a Single-Task (ST) architecture and a Multi-Task (MT) architecture. While the ST architecture models single emotional perception each time, the MT architecture jointly models every single annotation and the aggregated annotations at once. Our results show that the MT approach can more accurately model every single annotation and the aggregated annotations when compared to methods that are directly trained on the aggregated annotations. Furthermore, the MT approach achieves state-of-the-art results on the COGNIMUSE, IEMOCAP, and SemEval_2007 benchmarks.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Modeling Subjective Affect Annotations with Multi-Task Learning

Hayat

Ventura

Lapedriza

2022

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

Exploring contactless techniques in multimodal emotion recognition: insights into diverse applications, challenges, solutions, and prospects

Khan,

Xu,

Liu

et al. 2024

Multimedia Systems

View full text Add to dashboard Cite

In recent years, emotion recognition has received significant attention, presenting a plethora of opportunities for application in diverse fields such as human–computer interaction, psychology, and neuroscience, to name a few. Although unimodal emotion recognition methods offer certain benefits, they have limited ability to encompass the full spectrum of human emotional expression. In contrast, Multimodal Emotion Recognition (MER) delivers a more holistic and detailed insight into an individual's emotional state. However, existing multimodal data collection approaches utilizing contact-based devices hinder the effective deployment of this technology. We address this issue by examining the potential of contactless data collection techniques for MER. In our tertiary review study, we highlight the unaddressed gaps in the existing body of literature on MER. Through our rigorous analysis of MER studies, we identify the modalities, specific cues, open datasets with contactless cues, and unique modality combinations. This further leads us to the formulation of a comparative schema for mapping the MER requirements of a given scenario to a specific modality combination. Subsequently, we discuss the implementation of Contactless Multimodal Emotion Recognition (CMER) systems in diverse use cases with the help of the comparative schema which serves as an evaluation blueprint. Furthermore, this paper also explores ethical and privacy considerations concerning the employment of contactless MER and proposes the key principles for addressing ethical and privacy concerns. The paper further investigates the current challenges and future prospects in the field, offering recommendations for future research and development in CMER. Our study serves as a resource for researchers and practitioners in the field of emotion recognition, as well as those intrigued by the broader outcomes of this rapidly progressing technology.

show abstract

Recognizing Emotions evoked by Movies using Multitask Learning

Cited by 2 publications

References 34 publications

Modeling Subjective Affect Annotations with Multi-Task Learning

Modeling Subjective Affect Annotations with Multi-Task Learning

Exploring contactless techniques in multimodal emotion recognition: insights into diverse applications, challenges, solutions, and prospects

Contact Info

Product

Resources

About