Transformer-based Non-Verbal Emotion Recognition

Vaiani, Lorenzo; Koudounas, Alkis; Quatra, Moreno La; Cagliero, Luca; Garza, Paolo; Baralis, Elena

doi:10.1145/3551876.3554801

Cited by 3 publications

(4 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Baseline (FAU) [1] .2840 (.2828 ± .0016) .2801 (.2777 ± .0017) Baseline (VGGFACE 2) [1] .2488 (.2441 ± .0027) .1830 (.1985 ± .0088) Resnet-18 [14] .3893 (--) --(--) Former-DFER+MLGCN [16] .3454 (--) --(--) ViPER [15] .2978 (--) .2859 (--) FaceRNET [6] .3590 (--) .3607 (--)…”

Section: Videounclassified

See 1 more Smart Citation

ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges

Kollias¹,

Tzirakis²,

Baird³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

Section: Videounclassified

“…Baseline [1] .2382 (.2350 ± 0.0016) .2029 (.2014 ± .0086) Resnet-18 + DEEPSPECTRUM [14] .3968 (--) --(--) ViPER [15] .3025 (--) .2970 (--)…”

Section: Multimodalunclassified

ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges

Kollias¹,

Tzirakis²,

Baird³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…As an alternative vision-based strategy, we employ the DINO-trained ViT , which has been pretrained on the ImageNet-1K dataset in a self-supervised manner using the self-distillation with no labels (DINO) method [14]. This model has demonstrated its efficacy for various image-based tasks, including emotion recognition from facial expressions [16,58]. The model processes the extracted facial images and outputs a 384dimensional embedding for each image.…”

Section: Vision Transformer (Vit )mentioning

confidence: 99%

The MuSe 2022 Multimodal Sentiment Analysis Challenge

Christ

Amiriparian

Baird³

et al. 2022

Proceedings of the 3rd International on Multimodal Sentiment Analysis Workshop and Challenge

View full text Add to dashboard Cite

nnnnnnn feature sets. As a competitive baseline system, a Gated Recurrent Unit (GRU)-Recurrent Neural Network (RNN) is employed. On the respective sub-challenges' test datasets, it achieves a mean (across three continuous intensity targets) Pearson's Correlation Coefficient of .4727 for MuSe-Mimic, an Area Under the Curve (AUC) value of .8310 for MuSe-Humour and Concordance Correlation Coefficient (CCC) values of .7482 for arousal and .7827 for valence in the MuSe-Personalisation sub-challenge. CCS CONCEPTS• Computing methodologies → Neural networks; Artificial intelligence; Computer vision; Natural language processing.

show abstract

“…Former-DFER+MLGCN [32] also proposes an end-to-end network framework that has achieved remarkable performance on the Hume-Reaction dataset. To better utilize multimodal information, ViPER [30] proposed a multimodal architecture for emotion recognition which achieves excellent performance on the ERI task as well.…”

Section: Emotional Reaction Intensitymentioning

confidence: 99%

Hyperspectral Image Classification for Foreign Bodies in Medicine Based on Spatial-Spectral Fusion Feature

Zhu

Yin

et al. 2021

2021 China Automation Congress (CAC)

View full text Add to dashboard Cite

In this paper, we present our advanced solutions to the two sub-challenges of Affective Behavior Analysis in the wild (ABAW) 2023: the Emotional Reaction Intensity (ERI) Estimation Challenge and Expression (Expr) Classification Challenge. ABAW 2023 aims to tackle the challenge of affective behavior analysis in natural contexts, with the ultimate goal of creating intelligent machines and robots that possess the ability to comprehend human emotions, feelings, and behaviors. For the Expression Classification Challenge, we propose a streamlined approach that handles the challenges of classification effectively. However, our main contribution lies in our use of diverse models and tools to extract multimodal features such as audio and video cues from the Hume-Reaction dataset. By studying, analyzing, and combining these features, we significantly enhance the model's accuracy for sentiment prediction in a multimodal context. Furthermore, our method achieves outstanding results on the Emotional Reaction Intensity (ERI) Estimation Challenge, surpassing the baseline method by an impressive 84% increase, as measured by the Pearson Coefficient, on the validation dataset.

show abstract

Transformer-based Non-Verbal Emotion Recognition

Cited by 3 publications

References 20 publications

ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges

ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges

The MuSe 2022 Multimodal Sentiment Analysis Challenge

Hyperspectral Image Classification for Foreign Bodies in Medicine Based on Spatial-Spectral Fusion Feature

Contact Info

Product

Resources

About