Fusing Aligned and Non-aligned Face Information for Automatic Affect Recognition in the Wild: A Deep Learning Approach

Kim, Bo-Kyeong; Dong, Suh-Yeon; Roh, Jihyeon; Kim, Geonmin; Lee, Sang Yup

doi:10.1109/cvprw.2016.187

Cited by 92 publications

(43 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The performances using this dataset do not exceed 75.2%. This accuracy has been achieved by Pramerdorfer and Kampel [22], while other researches reached 71.2% using linear support vector machines [23], 73.73% by fusing aligned and non-aligned face information for automatic affect recognition in the wild [24], 70.66% using an end-to-end deep learning framework, based on the attentional convolutional network [25], 71.91% using three subnetworks with different depths [26] and 73.4 using a hybrid CNN-scale invariant feature transform aggregator [27].…”

Section: Introductionmentioning

confidence: 87%

“…Pramerdorfer et al [22] 75.2 Network Ensemble -SGD Tang [23] 71.2 Loss layer -SGD Kim et al [24] 73.73 Network Ensemble Intraface SGD Minae et al [25] 70.02 Loss layer Adam Hua et al [26] 71.91 Ensemble Network Straightforward Adam Connie [27] 73. In Figure 12 the manner the VGG model learns patterns for a given input image is presented.…”

Section: Cnn Model Accuracy [%] Cnn Type Preprocessing Optimizermentioning

confidence: 99%

See 1 more Smart Citation

Facial Expressions Recognition for Human–Robot Interaction Using Deep Convolutional Neural Networks with Rectified Adam Optimizer

Melinte

Vlădăreanu

2020

Sensors

View full text Add to dashboard Cite

The interaction between humans and an NAO robot using deep convolutional neural networks (CNN) is presented in this paper based on an innovative end-to-end pipeline method that applies two optimized CNNs, one for face recognition (FR) and another one for the facial expression recognition (FER) in order to obtain real-time inference speed for the entire process. Two different models for FR are considered, one known to be very accurate, but has low inference speed (faster region-based convolutional neural network), and one that is not as accurate but has high inference speed (single shot detector convolutional neural network). For emotion recognition transfer learning and fine-tuning of three CNN models (VGG, Inception V3 and ResNet) has been used. The overall results show that single shot detector convolutional neural network (SSD CNN) and faster region-based convolutional neural network (Faster R-CNN) models for face detection share almost the same accuracy: 97.8% for Faster R-CNN on PASCAL visual object classes (PASCAL VOCs) evaluation metrics and 97.42% for SSD Inception. In terms of FER, ResNet obtained the highest training accuracy (90.14%), while the visual geometry group (VGG) network had 87% accuracy and Inception V3 reached 81%. The results show improvements over 10% when using two serialized CNN, instead of using only the FER CNN, while the recent optimization model, called rectified adaptive moment optimization (RAdam), lead to a better generalization and accuracy improvement of 3%-4% on each emotion recognition CNN.

show abstract

Section: Introductionmentioning

confidence: 87%

Section: Cnn Model Accuracy [%] Cnn Type Preprocessing Optimizermentioning

confidence: 99%

Facial Expressions Recognition for Human–Robot Interaction Using Deep Convolutional Neural Networks with Rectified Adam Optimizer

Melinte

Vlădăreanu

2020

Sensors

View full text Add to dashboard Cite

show abstract

“…Accuracy Hand-crafted feature guided CNN [36] 61.86 AlexNet [37] 64.8 DNNRL [37] 70.6 ResNet [38] 72.4 VGG [38] 72.7 Ensemble of deep networks [39] 73.31 Alignment mapping networks + ensemble [39] 73.73 Single CNN [40] 71.47 Ensemble CNN [40] 73.73 Proposed 73.58…”

Section: Methodsmentioning

confidence: 99%

A Weakly Supervised learning technique for classifying facial expressions

Happy

Dantcheva

Brémond

2019

Pattern Recognition Letters

View full text Add to dashboard Cite

The universal hypothesis suggests that the six basic emotions-anger, disgust, fear, happiness, sadness, and surprise-are being expressed by similar facial expressions by all humans. While existing datasets support the universal hypothesis and comprise of images and videos with discrete disjoint labels of profound emotions, real-life data contains jointly occurring emotions and expressions of different intensities. Models, which are trained using categorical one-hot vectors often over-fit and fail to recognize low or moderate expression intensities. Motivated by the above, as well as by the lack of sufficient annotated data, we here propose a weakly supervised learning technique for expression classification, which leverages the information of unannotated data. Crucial in our approach is that we first train a convolutional neural network (CNN) with label smoothing in a supervised manner and proceed to tune the CNN-weights with both labelled and unlabelled data simultaneously. Experiments on four datasets demonstrate large performance gains in cross-database performance, as well as show that the proposed method achieves to learn different expression intensities, even when trained with categorical samples.

show abstract

“…Inspired by the center loss [168], which penalizes the distance between deep features and their corresponding class centers, two variations were proposed to assist the supervision of the softmax loss for more discriminative features for FER: (1) island loss [140] was formalized to further increase the pairwise distances between different class centers (see Fig. 6(b)), and (2) locality-preserving loss (LP loss) [44] was [76], [146], [173] Simple Average determine the class with the highest mean score using the posterior class probabilities yielded from each individual with the same weight [76], [146], [173] Weighted Average determine the class with the highest weighted mean score using the posterior class probabilities yielded from each individual with different weights [57], [78], [147], [153] formalized to pull the locally neighboring features of the same class together so that the intra-class local clusters of each class are compact. Besides, based on the triplet loss [169], which requires one positive example to be closer to the anchor than one negative example with a fixed gap, two variations were proposed to replace or assist the supervision of the softmax loss: (1) exponential triplet-based loss [145] was formalized to give difficult samples more weight when updating the network, and (2) (N+M)-tuples cluster loss [77] was formalized to alleviate the difficulty of anchor selection and threshold validation in the triplet loss for identityinvariant FER (see Fig.…”

Section: Auxiliary Blocks and Layersmentioning

confidence: 99%

Deep Facial Expression Recognition: A Survey

Deng

2022

IEEE Trans. Affective Comput.

1,082

593

View full text Add to dashboard Cite

With the transition of facial expression recognition (FER) from laboratory-controlled to challenging in-the-wild conditions and the recent success of deep learning techniques in various fields, deep neural networks have increasingly been leveraged to learn discriminative representations for automatic FER. Recent deep FER systems generally focus on two important issues: overfitting caused by a lack of sufficient training data and expression-unrelated variations, such as illumination, head pose and identity bias. In this paper, we provide a comprehensive survey on deep FER, including datasets and algorithms that provide insights into these intrinsic problems. First, we introduce the available datasets that are widely used in the literature and provide accepted data selection and evaluation principles for these datasets. We then describe the standard pipeline of a deep FER system with the related background knowledge and suggestions of applicable implementations for each stage. For the state of the art in deep FER, we review existing novel deep neural networks and related training strategies that are designed for FER based on both static images and dynamic image sequences, and discuss their advantages and limitations. Competitive performances on widely used benchmarks are also summarized in this section. We then extend our survey to additional related issues and application scenarios. Finally, we review the remaining challenges and corresponding opportunities in this field as well as future directions for the design of robust deep FER systems.

show abstract

Fusing Aligned and Non-aligned Face Information for Automatic Affect Recognition in the Wild: A Deep Learning Approach

Cited by 92 publications

References 32 publications

Facial Expressions Recognition for Human–Robot Interaction Using Deep Convolutional Neural Networks with Rectified Adam Optimizer

Facial Expressions Recognition for Human–Robot Interaction Using Deep Convolutional Neural Networks with Rectified Adam Optimizer

A Weakly Supervised learning technique for classifying facial expressions

Deep Facial Expression Recognition: A Survey

Contact Info

Product

Resources

About