A Multi-task Learning Framework for Time-continuous Emotion Estimation from Crowd Annotations

Abadi, Mojtaba Khomami; Abad, Azad; Subramanian, Ramanathan; Rostamzadeh, Negar; Ricci, Elisa; Varadarajan, Jagannadan; Sebe, Nicu

doi:10.1145/2660114.2660116

Cited by 5 publications

(4 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Implementation details are given next. 1. Chapter 6 of [8] presented an aspect model with Bayesian active learning algorithm for regression problems and applied it to two movie rating applications, but it is single-task active learning instead of multi-task active learning.…”

Section: Single-task Gsymentioning

confidence: 99%

“…Su et al [21] proposed MTL with low rank attribute embedding to perform person re-identification on multi-cameras, and demonstrated that it significantly outperformed existing single-task and multi-task approaches. Abadi et al [1] proposed MTL-based regression models to simultaneously learn the relationship between low-level audio-visual features and highlevel valence/arousal ratings from a collection of movie scenes. They can better predict valence and arousal ratings than scenespecific models.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Affect Estimation in 3D Space Using Multi-Task Active Learning for Regression

Huang

2022

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

Acquisition of labeled training samples for affective computing is usually costly and time-consuming, as affects are intrinsically subjective, subtle and uncertain, and hence multiple human assessors are needed to evaluate each affective sample. Particularly, for affect estimation in the 3D space of valence, arousal and dominance, each assessor has to perform the evaluations in three dimensions, which makes the labeling problem even more challenging. Many sophisticated machine learning approaches have been proposed to reduce the data labeling requirement in various other domains, but so far few have considered affective computing. This paper proposes two multi-task active learning for regression approaches, which select the most beneficial samples to label, by considering the three affect primitives simultaneously. Experimental results on the VAM corpus demonstrated that our optimal sample selection approaches can result in better estimation performance than random selection and several traditional single-task active learning approaches. Thus, they can help alleviate the data labeling problem in affective computing, i.e., better estimation performance can be obtained from fewer labeling queries.

show abstract

Section: Single-task Gsymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Affect Estimation in 3D Space Using Multi-Task Active Learning for Regression

Huang

2022

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

show abstract

“…This paper is an extension of the work presented in [2], where Multi-task learning (MTL) was employed to learn a crowd-based model for predicting dynamic arousal (A) and valence (V) levels in movie snippets. Given a set of related tasks, MTL simultaneously learns all tasks by modeling the similarities as well as differences among them to build task-specific classification or regression models.…”

Section: Introductionmentioning

confidence: 99%

“…In this work, we seek to improve the efficacy of crowdbased models by employing a small amount of expert data to guide the learning process-we employed 16 experts familiar with emotional attributes to provide dynamic A,V ratings for the clips annotated by crowdworkers 2 , and enhance crowd models employing expert knowledge via a novel expert-guided MTL (EG-MTL) algorithm. The EG-MTL algorithm seeks to simultaneously minimize the loss with respect to both crowd and expert labels in the optimization framework and learns a set of weights corresponding to each of the movie clips for which crowd annotations were sought.…”

Section: Introductionmentioning

confidence: 99%

Expert and Crowd-Guided Affect Annotation and Prediction

Subramanian¹,

Yan²,

Sebe³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We employ crowdsourcing to acquire time-continuous affective annotations for movie clips, and refine noisy models trained from these crowd annotations incorporating expert information within a Multi-task Learning (MTL) framework. We propose a novel expert guided MTL (EG-MTL) algorithm, which minimizes the loss with respect to both crowd and expert labels to learn a set of weights corresponding to each movie clip for which crowd annotations are acquired. We employ EG-MTL to solve two problems, namely, P1: where dynamic annotations acquired from both experts and crowdworkers for the Validation set are used to train a regression model with audio-visual clip descriptors as features, and predict dynamic arousal and valence levels on 5-15 second snippets derived from the clips; and P2: where a classification model trained on the Validation set using dynamic crowd and expert annotations (as features) and static affective clip labels is used for binary emotion recognition on the Evaluation set for which only dynamic crowd annotations are available. Observed experimental results confirm the effectiveness of the EG-MTL algorithm, which is reflected via improved arousal and valence estimation for P1, and higher recognition accuracy for P2.

show abstract

Crowdsourcing Empathetic Intelligence

Katsimerou

Albeda

Huldtgren

et al. 2016

ACM Trans. Intell. Syst. Technol.

View full text Add to dashboard Cite

Unobtrusive recognition of the user's mood is an essential capability for affect-adaptive systems. Mood is a subtle, long-term affective state, often misrecognized even by humans. The challenge to train a machine to recognize it from, for example, a video of the user, is significant, and already begins with the lack of ground truth for supervised learning. Existing affective databases consist mainly of short videos, annotated in terms of expressed emotions rather than mood. In very few cases, we encounter perceived mood annotations, of questionable reliability, however, due to the subjectivity of mood estimation and the small number of coders involved. In this work, we introduce a new database for mood recognition from video. Our database contains 180 long, acted videos, depicting typical daily scenarios, and subtle facial and bodily expressions. The videos cover three visual modalities (face, body, Kinect data), and are annotated in terms of emotions (via G-trace) and mood (via the Self-Assessment Manikin and the AffectButton). To annotate the database exhaustively, we exploit crowdsourcing to reach out to an extensive number of nonexpert coders. We validate the reliability of our crowdsourced annotations by (1) adopting a number of criteria to filter out unreliable coders, and (2) comparing the annotations of a subset of our videos with those collected in a controlled lab setting.

show abstract

A Multi-task Learning Framework for Time-continuous Emotion Estimation from Crowd Annotations

Cited by 5 publications

References 33 publications

Affect Estimation in 3D Space Using Multi-Task Active Learning for Regression

Affect Estimation in 3D Space Using Multi-Task Active Learning for Regression

Expert and Crowd-Guided Affect Annotation and Prediction

Crowdsourcing Empathetic Intelligence

Contact Info

Product

Resources

About