Temporal Stochastic Softmax for 3D CNNs: An Application in Facial Expression Recognition

Ayral, Théo; Pedersoli, Marco; Bacon, Simon L.; Granger, Éric

doi:10.1109/wacv48630.2021.00307

Cited by 11 publications

(5 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some later studies [18,19] further proved that the models using the CNN-RNN scheme are practical spatial-temporal feature extractors. In addition, independent 3D CNN architecture and its variants also performed competitively in video-based FER tasks [20,21].…”

Section: Video-based Facial Expression Recognitionmentioning

confidence: 99%

Self-Supervised Facial Motion Representation Learning via Contrastive Subclips

et al. 2023

View full text Add to dashboard Cite

Facial motion representation learning has become an exciting research topic, since biometric technologies are becoming more common in our daily lives. One of its applications is identity verification. After recording a dynamic facial motion video for enrollment, the user needs to show a matched facial appearance and make a facial motion the same as the enrollment for authentication. Some recent research papers have discussed the benefits of this new biometric technology and reported promising results for both static and dynamic facial motion verification tasks. Our work extends the existing approaches and introduces compound facial actions, which contain more than one dominant facial action in one utterance. We propose a new self-supervised pretraining method called contrastive subclips that improves the model performance with these more complex and secure facial motions. The experimental results show that the contrastive subclips method improves upon the baseline approaches, and the model performance for test data can reach 89.7% average precision.

show abstract

Section: Video-based Facial Expression Recognitionmentioning

confidence: 99%

Self-Supervised Facial Motion Representation Learning via Contrastive Subclips

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Although these methods perform well by selecting peak frames, they neglect the temporal dynamics and correlation among facial frames. Different from static frame-based methods, dynamic sequence based methods usually use 3D convolution neural networks (3DCNN) [7], long-short term memory (LSTM) [8] to learn the spatiotemporal relationships, which can model long-term dependencies and improve the performance of DFER. The performances of these methods are still far from being satisfactory, because of occlusions, variant head poses, poor illumination and other unexpected issues in real-world scenes.…”

Section: Introductionmentioning

confidence: 99%

Logo-Former: Local-Global Spatio-Temporal Transformer for Dynamic Facial Expression Recognition

Sun

2023

ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Previous methods for dynamic facial expression recognition (DFER) in the wild are mainly based on Convolutional Neural Networks (CNNs), whose local operations ignore the longrange dependencies in videos. Transformer-based methods for DFER can achieve better performances but result in higher FLOPs and computational costs. To solve these problems, the local-global spatio-temporal Transformer (LOGO-Former) is proposed to capture discriminative features within each frame and model contextual relationships among frames while balancing the complexity. Based on the priors that facial muscles move locally and facial expressions gradually change, we first restrict both the space attention and the time attention to a local window to capture local interactions among feature tokens. Furthermore, we perform the global attention by querying a token with features from each local window iteratively to obtain long-range information of the whole video sequence. In addition, we propose the compact loss regularization term to further encourage the learned features have the minimum intra-class distance and the maximum inter-class distance. Experiments on two in-the-wild dynamic facial expression datasets (i.e., DFEW and FERV39K) indicate that our method provides an effective way to make use of the spatial and temporal dependencies for DFER.

show abstract

“…Different from static frame-based methods, dynamic sequence based methods usually use 3D convolution neural networks (3DCNN) [2], long-short term memory (LSTM) [54] to learn the spatio-temporal relationships, which can model long-term dependencies and improve the performance of DFER. For instance, Kim et al [23] propose to use a LSTM network to learn the temporal characteristics of the spatial features, which achieves higher recognition rates.…”

Section: Introductionmentioning

confidence: 99%

“…Although these methods employ the 3D CNN to capture the temporal correlation, they only use one or two layers with 3D convolutional filters and fail to model the complicated spatio-temporal correlation. Very recently, Ayral et al [2] propose to re-weight different clips and achieve a clip-aware 3D CNN for dynamic facial expression, which outperforms other methods on AFEW. A novel framework called EC-STFL [21] is also proposed to reduce the inter-class distance and enhance the intraclass correlation.…”

Section: Introductionmentioning

confidence: 99%

Spatio-Temporal Transformer for Dynamic Facial Expression Recognition in the Wild

Ma¹,

Sun²,

Li³

2022

Preprint

View full text Add to dashboard Cite

Previous methods for dynamic facial expression in the wild are mainly based on Convolutional Neural Networks (CNNs), whose local operations ignore the long-range dependencies in videos. To solve this problem, we propose the spatio-temporal Transformer (STT) to capture discriminative features within each frame and model contextual relationships among frames. Spatio-temporal dependencies are captured and integrated by our unified Transformer. Specifically, given an image sequence consisting of multiple frames as input, we utilize the CNN backbone to translate each frame into a visual feature sequence. Subsequently, the spatial attention and the temporal attention within each block are jointly applied for learning spatio-temporal representations at the sequence level. In addition, we propose the compact softmax cross entropy loss to further encourage the learned features have the minimum intra-class distance and the maximum inter-class distance. Experiments on two in-the-wild dynamic facial expression datasets (i.e., DFEW and AFEW) indicate that our method provides an effective way to make use of the spatial and temporal dependencies for dynamic facial expression recognition. The source code and the training logs will be made publicly available. CCS CONCEPTS• Computing methodologies → Activity recognition and understanding; Supervised learning.

show abstract

Temporal Stochastic Softmax for 3D CNNs: An Application in Facial Expression Recognition

Cited by 11 publications

References 44 publications

Self-Supervised Facial Motion Representation Learning via Contrastive Subclips

Self-Supervised Facial Motion Representation Learning via Contrastive Subclips

Logo-Former: Local-Global Spatio-Temporal Transformer for Dynamic Facial Expression Recognition

Spatio-Temporal Transformer for Dynamic Facial Expression Recognition in the Wild

Contact Info

Product

Resources

About