Recognition of identity and expressions as integrated processes

Abstract

According to the dominant account of face processing, recognition of emotional expressions is implemented by the superior temporal sulcus (STS), while recognition of face identity is implemented by inferior temporal cortex (IT) (Haxby et al., 2000). However, recent patient and imaging studies (Fox et al., 2011, Anzellotti et al. 2017) found that the STS also encodes information about identity.Jointly representing expression and identity might be computationally advantageous: learning to recognize expressions could lead to the emergence of representations that support identity recognition. To test this hypothesis, we trained a deep densely connected convolutional network (DenseNet, Huang et al., 2017) to classify face images from the fer2013 dataset as either angry, disgusted, afraid, happy, sad, surprised, or neutral. We then froze the weights of the DenseNet and trained linear layers attached to progressively deeper layers of this net to classify either emotion or identity using a subset of the Karolinska (KDEF) dataset. Finally, we tested emotion and identity classification in left out images in the KDEF dataset that were not used for training.Classification accuracy for emotions in the KDEF dataset increased from early to late layers of the DenseNet, indicating successful transfer across datasets. Critically, classification accuracy for identity also increased from early to late layers of this DenseNet, despite the fact that it had not been trained to classify identity. A linear layer trained on the DenseNet features vastly outperformed a linear layer trained on pixels (98.8% vs 68.7%), demonstrating that the high accuracy obtained with the DenseNet features cannot be explained by low-level confounds. These results show that learning to recognize facial expressions can lead to the spontaneous emergence of representations that support the recognition of identity, thus offering a principled computational account for the discovery of expression and identity representations within the same portion of STS.

Emerged human-like facial expression representation in a deep convolutional neural network

“…1C). The results indicated that the expression-selective units spontaneously emerged in the VGG-Face pretrained for face identity recognition, which echoed previous findings (19,20).…”

“…Thus, DCNNs could be a useful model simulating the processes of biological neural systems. More recently, several seminal studies have found that the DCNNs trained to recognize facial expression spontaneously developed facial identity recognition ability, and vice versa, suggesting that integrated representations of identity and expression may arise naturally within neural networks like humans do (19,20). However, a recent study found that face identity-selective units could spontaneously emerge in an untrained DCNN (21), which seemed to cast substantial doubt on the role of nurture in developing face perception and the abovementioned speculation.…”

Convolutional neural network hyperparameters for face emotion recognition using genetic algorithm

“…Therefore, hyperparameter optimization via the CNN-GA approach offers promising prospects for improving model capabilities in increasingly complex recognition tasks in the future. Tang's network structure 71.2 [37] Caffe-imageNet 65.5 [22] MNF CNN+L2 SVM 70.3 [38] Raspberry Pi 65.97 [39] DenseNet 63.50 [40] VGG16 69.40 [41] Attention CNN 70.02 [42] ResNet with gate implementation 71.80 [43] VGGNet 73.28 [44] VGG progressive SpinalNet 74.39 VGG SpinalNet 74.45 [45] Mini-Xception 66.00 [46] CNN with transfer learning 72.00 [47] CNN with HOG feature 75.…”