POSTER: A Pyramid Cross-Fusion Transformer Network for Facial Expression Recognition

Zheng, Chuansheng; Mendieta, Matías; Chen, Chen

doi:10.48550/arxiv.2204.04083

Cited by 6 publications

(11 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We evaluat the FER performance of POSTER V2 on the widely used RAF-DB, AffectNet and CAER-S Settings. Similar to POSTER V1 [58], we also use the ir50 [7] network pre-trained on the Ms-Celeb-1M [14] dataset as the image backbone. And MobileFaceNet [2] with frozen weights is used as our facial landmark detector.…”

Section: Experiments Setupmentioning

confidence: 99%

See 1 more Smart Citation

POSTER++: A simpler and stronger facial expression recognition network

Mao¹,

Xu²,

Yin³

et al. 2023

Preprint

View full text Add to dashboard Cite

Facial expression recognition (FER) plays an important role in a variety of real-world applications such as human-computer interaction. POSTER V1 achieves the state-of-the-art (SOTA) performance in FER by effectively combining facial landmark and image features through two-stream pyramid cross-fusion design. However, the architecture of POSTER V1 is undoubtedly complex. It causes expensive computational costs. In order to relieve the computational pressure of POSTER V1, in this paper, we propose POSTER V2. It improves POSTER V1 in three directions: cross-fusion, two-stream, and multiscale feature extraction. In cross-fusion, we use windowbased cross-attention mechanism replacing vanilla crossattention mechanism. We remove the image-to-landmark branch in the two-stream design. For multi-scale feature extraction, POSTER V2 combines images with landmark's multi-scale features to replace POSTER V1's pyramid design. Extensive experiments on several standard datasets show that our POSTER V2 achieves the SOTA FER performance with the minimum computational cost. For example, POSTER V2 reached 92.21% on RAF-DB, 67.49% on AffectNet (7 cls) and 63.77% on AffectNet (8 cls), respectively, using only 8.4G floating point operations (FLOPs) and 43.7M parameters (Param). This demonstrates the effectiveness of our improvements. The code and models are available at https://github.com/Talented-Q/ POSTER_V2.

show abstract

Section: Experiments Setupmentioning

confidence: 99%

“…Among many excellent FER works, POSTER V1 [58] stands out with state-of-the-art (SOTA) performance. POSTER V1 mainly solves three key issues of FER at the same time: inter-class similarity, intra-class discrepancy and scale sensitivity.…”

Section: Introductionmentioning

confidence: 99%

POSTER++: A simpler and stronger facial expression recognition network

Mao¹,

Xu²,

Yin³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…POSTER:The two-stream Pyramid crOss-fuSion Trans-formER network (POSTER) (Zheng, Mendieta, and Chen 2022) is proposed to address the challenges of facial expression recognition. It effectively integrates facial landmark and direct image features using a transformer-based crossfusion paradigm and employs a pyramid structure to ensure scale invariance.…”

Section: Visual Featuresmentioning

confidence: 99%

Facial Affect Recognition based on Transformer Encoder and Audiovisual Fusion for the ABAW5 Challenge

Zhang¹,

An²,

Zishun³

et al. 2023

Preprint

View full text Add to dashboard Cite

In this paper, we present our solutions for the 5th Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW), which includes four sub-challenges of Valence-Arousal (VA) Estimation, Expression (Expr) Classification, Action Unit (AU) Detection and Emotional Reaction Intensity (ERI) Estimation. The 5th ABAW competition focuses on facial affect recognition utilizing different modalities and datasets. In our work, we extract powerful audio and visual features using a large number of sota models. These features are fused by Transformer Encoder and TEMMA. Besides, to avoid the possible impact of large dimensional differences between various features, we design an Affine Module to align different features to the same dimension. Extensive experiments demonstrate that the superiority of the proposed method. For the VA Estimation sub-challenge, our method obtains the mean Concordance Correlation Coefficient (CCC) of 0.6066. For the Expression Classification subchallenge, the average F1 Score is 0.4055. For the AU Detection sub-challenge, the average F1 Score is 0.5296. For the Emotional Reaction Intensity Estimation sub-challenge, the average pearson's correlations coefficient on the validation set is 0.3968. All of the results of four sub-challenges outperform the baseline with a large margin.

show abstract

“…In 2020, Dosovitskiy et al [ 25 ] proposed a new ViT that trained the model on image classification tasks (e.g., FER) and demonstrated promising results. Since then, numerous Transformer-based models designed for vision-related tasks have garnered significant attention from researchers [ 43 , 44 , 45 , 46 ]. These Transformer-based models split an image into tokens.…”

Section: Related Workmentioning

confidence: 99%

VaBTFER: An Effective Variant Binary Transformer for Facial Expression Recognition

Shen,

Jin

2023

Sensors

View full text Add to dashboard Cite

Existing Transformer-based models have achieved impressive success in facial expression recognition (FER) by modeling the long-range relationships among facial muscle movements. However, the size of pure Transformer-based models tends to be in the million-parameter level, which poses a challenge for deploying these models. Moreover, the lack of inductive bias in Transformer usually leads to the difficulty of training from scratch on limited FER datasets. To address these problems, we propose an effective and lightweight variant Transformer for FER called VaTFER. In VaTFER, we firstly construct action unit (AU) tokens by utilizing action unit-based regions and their histogram of oriented gradient (HOG) features. Then, we present a novel spatial-channel feature relevance Transformer (SCFRT) module, which incorporates multilayer channel reduction self-attention (MLCRSA) and a dynamic learnable information extraction (DLIE) mechanism. MLCRSA is utilized to model long-range dependencies among all tokens and decrease the number of parameters. DLIE’s goal is to alleviate the lack of inductive bias and improve the learning ability of the model. Furthermore, we use an excitation module to replace the vanilla multilayer perception (MLP) for accurate prediction. To further reduce computing and memory resources, we introduce a binary quantization mechanism, formulating a novel lightweight Transformer model called variant binary Transformer for FER (VaBTFER). We conduct extensive experiments on several commonly used facial expression datasets, and the results attest to the effectiveness of our methods.

show abstract

POSTER: A Pyramid Cross-Fusion Transformer Network for Facial Expression Recognition

Cited by 6 publications

References 32 publications

POSTER++: A simpler and stronger facial expression recognition network

POSTER++: A simpler and stronger facial expression recognition network

Facial Affect Recognition based on Transformer Encoder and Audiovisual Fusion for the ABAW5 Challenge

VaBTFER: An Effective Variant Binary Transformer for Facial Expression Recognition

Contact Info

Product

Resources

About