Adversarial-based neural networks for affect estimations in the wild

Aspandi, Decky; Mallol-Ragolta, Adria; Schuller, Björn W.; Binefa, Xavier

doi:10.48550/arxiv.2002.00883

Cited by 3 publications

(4 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The EmoFAN deep learning model [1] jointly predicted discrete emotional states and continuous affect dimensions by building upon the face alignment network proposed in [48], thereby achieving the best performance on the AfewVA dataset [49]. Aspandi et al [50] estimated affect in the wild by exploiting adversarial neural networks that build high-level representations of audiovisual information. Parthasarathy and Sundaram [51] demonstrated that multimodal deep learning affect models can significantly improve affect detection in the wild.…”

Section: Affect Modeling In the Wildmentioning

confidence: 99%

From the Lab to the Wild: Affect Modeling Via Privileged Information

Makantasis

Pinitas²,

Liapis³

et al. 2024

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

How can we reliably transfer affect models trained in controlled laboratory conditions (in-vitro) to uncontrolled real-world settings (in-vivo)? The information gap between in-vitro and in-vivo applications defines a core challenge of affective computing. This gap is caused by limitations related to affect sensing including intrusiveness, hardware malfunctions and availability of sensors. As a response to these limitations, we introduce the concept of privileged information for operating affect models in real-world scenarios (in the wild). Privileged information enables affect models to be trained across multiple modalities available in a lab, and ignore, without significant performance drops, those modalities that are not available when they operate in the wild. Our approach is tested in two multimodal affect databases one of which is designed for testing models of affect in the wild. By training our affect models using all modalities and then using solely raw footage frames for testing the models, we reach the performance of models that fuse all available modalities for both training and testing. The results are robust across both classification and regression affect modeling tasks which are dominant paradigms in affective computing. Our findings make a decisive step towards realizing affect interaction in the wild.

show abstract

Section: Affect Modeling In the Wildmentioning

confidence: 99%

From the Lab to the Wild: Affect Modeling Via Privileged Information

Makantasis

Pinitas²,

Liapis³

et al. 2024

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

show abstract

“…EmoFAN achieves the best performance on the AfewVA dataset [31]. Aspandi et al [32] use adversarial-based neural networks to learn latent representations from audiovisual signals for estimating affect in the wild. The authors of [33] use multimodal transformers to capture and exploit temporal dynamics of audiovisual information towards detecting affect states.…”

Section: B Affect Modeling In the Wildmentioning

confidence: 99%

Privileged Information for Modeling Affect In The Wild

Makantasis¹,

Melhárt²,

Liapis³

et al. 2021

Preprint

View full text Add to dashboard Cite

A key challenge of affective computing research is discovering ways to reliably transfer affect models that are built in the laboratory to real world settings, namely in the wild. The existing gap between in vitro and in vivo affect applications is mainly caused by limitations related to affect sensing including intrusiveness, hardware malfunctions, availability of sensors, but also privacy and security. As a response to these limitations in this paper we are inspired by recent advances in machine learning and introduce the concept of privileged information for operating affect models in the wild. The presence of privileged information enables affect models to be trained across multiple modalities available in a lab setting and ignore modalities that are not available in the wild with no significant drop in their modeling performance. The proposed privileged information framework is tested in a game arousal corpus that contains physiological signals in the form of heart rate and electrodermal activity, game telemetry, and pixels of footage from two dissimilar games that are annotated with arousal traces. By training our arousal models using all modalities (in vitro) and using solely pixels for testing the models (in vivo), we reach levels of accuracy obtained from models that fuse all modalities both for training and testing. The findings of this paper make a decisive step towards realizing affect interaction in the wild.

show abstract

“…In the literature, estimation of activation, valence and dominance attributes is referred as Continuous Emotion Recognition (CER). Although CER is widely studied by the speech processing community [6], [7], [8], it has been studied over the visual channels [9], [10], [11]. In [9], a deep attention based convolutional network is proposed to detect facial expressions and emotions.…”

Section: Introductionmentioning

confidence: 99%

“…In [9], a deep attention based convolutional network is proposed to detect facial expressions and emotions. Aspandi et al [10] proposed an adversarial training approach to jointly estimate whether the image is fake or not while estimating the activation and valence (AV) attributes. In [11], VGG-16 driven visual features are used as the input of a stacked convolutional recurrent neural network (CRNN) for affect recognition in the wild.…”

Section: Introductionmentioning

confidence: 99%

Use of Affective Visual Information for Summarization of Human-Centric Videos

Köprü¹,

Erzin²

2021

Preprint

View full text Add to dashboard Cite

Increasing volume of user-generated human-centric video content and their applications, such as video retrieval and browsing, require compact representations that are addressed by the video summarization literature. Current supervised studies formulate video summarization as a sequence-to-sequence learning problem and the existing solutions often neglect the surge of human-centric view, which inherently contains affective content. In this study, we investigate the affective-information enriched supervised video summarization task for human-centric videos. First, we train a visual input-driven state-of-the-art continuous emotion recognition model (CER-NET) on the RECOLA dataset to estimate emotional attributes. Then, we integrate the estimated emotional attributes and the high-level representations from the CER-NET with the visual information to define the proposed affective video summarization architectures (AVSUM). In addition, we investigate the use of attention to improve the AVSUM architectures and propose two new architectures based on temporal attention (TA-AVSUM) and spatial attention (SA-AVSUM). We conduct video summarization experiments on the TvSum database. The proposed AVSUM-GRU architecture with an early fusion of high level GRU embeddings and the temporal attention based TA-AVSUM architecture attain competitive video summarization performances by bringing strong performance improvements for the human-centric videos compared to the state-of-the-art in terms of F-score and self-defined face recall metrics.

show abstract

Adversarial-based neural networks for affect estimations in the wild

Cited by 3 publications

References 32 publications

From the Lab to the Wild: Affect Modeling Via Privileged Information

From the Lab to the Wild: Affect Modeling Via Privileged Information

Privileged Information for Modeling Affect In The Wild

Use of Affective Visual Information for Summarization of Human-Centric Videos

Contact Info

Product

Resources

About