Video-based Emotion Recognition Using Deeply-Supervised Neural Networks

Fan, Yingruo; Lam, Jacqueline C.K.; Li, Victor O. K.

doi:10.1145/3242969.3264978

Cited by 68 publications

(46 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fan et al [40], the performance of CAER-Net also will be highly boosted. We reserve this as further works.…”

Section: Results On the Afew Datasetmentioning

confidence: 98%

“…Unlike this, we focused on investigating how context information helps to improve the emotion recognition performance. For this purpose, we choice shallow architecture rather than Fan et al [40]. If the face encoding stream adopt more complicated networks such Table 6.…”

Section: Results On the Afew Datasetmentioning

confidence: 99%

“…Face encoding stream. As in existing facial expression analysis approaches [6,20,40], our networks also have the facial expression encoding module. We first detect and crop the facial regions using the off-the-shelf face detectors [41] to build input of face stream V F .…”

Section: Two-stream Encoding Networkmentioning

confidence: 99%

See 2 more Smart Citations

Context-Aware Emotion Recognition Networks

Lee

Kim

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

205

134

View full text Add to dashboard Cite

Traditional techniques for emotion recognition have focused on the facial expression analysis only, thus providing limited ability to encode context that comprehensively represents the emotional responses. We present deep networks for context-aware emotion recognition, called CAER-Net, that exploit not only human facial expression but also context information in a joint and boosting manner. The key idea is to hide human faces in a visual scene and seek other contexts based on an attention mechanism. Our networks consist of two sub-networks, including two-stream encoding networks to separately extract the features of face and context regions, and adaptive fusion networks to fuse such features in an adaptive fashion. We also introduce a novel benchmark for context-aware emotion recognition, called CAER, that is more appropriate than existing benchmarks both qualitatively and quantitatively. On several benchmarks, CAER-Net proves the effect of context for emotion recognition. Our dataset is available at

show abstract

“…Fan et al [40], the performance of CAER-Net also will be highly boosted. We reserve this as further works.…”

Section: Results On the Afew Datasetmentioning

confidence: 98%

Section: Results On the Afew Datasetmentioning

confidence: 99%

See 1 more Smart Citation

Context-Aware Emotion Recognition Networks

Lee

Kim

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

205

134

View full text Add to dashboard Cite

show abstract

“…For an exact and fair comparison, the numerical values of the conventional methods were quoted directly from previous studies [ 27 , 29 , 37 , 58 , 59 , 60 , 61 ]. CNN-RNN-based techniques [ 27 , 29 , 60 ] and 2D CNN-based ones [ 37 , 58 , 59 , 61 ] were examined. Note that [ 37 ] used five-fold cross-validation jointly with a training set and validation set, so it could not be fairly compared with the others.…”

Section: Resultsmentioning

confidence: 99%

Visual Scene-Aware Hybrid and Multi-Modal Feature Aggregation for Facial Expression Recognition

Lee

Kim

Song

2020

Sensors

View full text Add to dashboard Cite

Facial expression recognition (FER) technology has made considerable progress with the rapid development of deep learning. However, conventional FER techniques are mainly designed and trained for videos that are artificially acquired in a limited environment, so they may not operate robustly on videos acquired in a wild environment suffering from varying illuminations and head poses. In order to solve this problem and improve the ultimate performance of FER, this paper proposes a new architecture that extends a state-of-the-art FER scheme and a multi-modal neural network that can effectively fuse image and landmark information. To this end, we propose three methods. To maximize the performance of the recurrent neural network (RNN) in the previous scheme, we first propose a frame substitution module that replaces the latent features of less important frames with those of important frames based on inter-frame correlation. Second, we propose a method for extracting facial landmark features based on the correlation between frames. Third, we propose a new multi-modal fusion method that effectively fuses video and facial landmark information at the feature level. By applying attention based on the characteristics of each modality to the features of the modality, novel fusion is achieved. Experimental results show that the proposed method provides remarkable performance, with 51.4% accuracy for the wild AFEW dataset, 98.5% accuracy for the CK+ dataset and 81.9% accuracy for the MMI dataset, outperforming the state-of-the-art networks.

show abstract

“…Both [28] and [29] input two LBP maps and a gray image for CNN models. Deeplysupervised networks are used in [29] and [30], which add supervision on intermediate layers. For static methods, [31] gets slightly better performance than ours.…”

Section: Evaluation On Afew 80mentioning

confidence: 99%

Frame Attention Networks for Facial Expression Recognition in Videos

Meng

Peng

Wang

et al. 2019

2019 IEEE International Conference on Image Processing (ICIP)

162

View full text Add to dashboard Cite

The video-based facial expression recognition aims to classify a given video into several basic emotions. How to integrate facial features of individual frames is crucial for this task. In this paper, we propose the Frame Attention Networks (FAN) 1 , to automatically highlight some discriminative frames in an end-to-end framework. The network takes a video with a variable number of face images as its input and produces a fixed-dimension representation. The whole network is composed of two modules. The feature embedding module is a deep Convolutional Neural Network (CNN) which embeds face images into feature vectors. The frame attention module learns multiple attention weights which are used to adaptively aggregate the feature vectors to form a single discriminative video representation. We conduct extensive experiments on CK+ and AFEW8.0 datasets. Our proposed FAN shows superior performance compared to other CNN based methods and achieves state-of-the-art performance on CK+.

show abstract

Video-based Emotion Recognition Using Deeply-Supervised Neural Networks

Cited by 68 publications

References 17 publications

Context-Aware Emotion Recognition Networks

Context-Aware Emotion Recognition Networks

Visual Scene-Aware Hybrid and Multi-Modal Feature Aggregation for Facial Expression Recognition

Frame Attention Networks for Facial Expression Recognition in Videos

Contact Info

Product

Resources

About