Facial expression is retained in deep networks trained for face identification

Abstract

Facial expressions distort visual cues for identification in two-dimensional images. Face processing systems in the brain must decouple image-based information from multiple sources to operate in the social world. Deep convolutional neural networks (DCNN) trained for face identification retain identity-irrelevant, image-based information (e.g., viewpoint). We asked whether a DCNN trained for identity also retains expression information that generalizes over viewpoint change. DCNN representations were generated for a controlled dataset containing images of 70 actors posing 7 facial expressions (happy, sad, angry, surprised, fearful, disgusted, neutral), from 5 viewpoints (frontal, 90° and 45° left and right profiles). Two-dimensional visualizations of the DCNN representations revealed hierarchical groupings by identity, followed by viewpoint, and then by facial expression. Linear discriminant analysis of full-dimensional representations predicted expressions accurately, mean 76.8% correct for happiness, followed by surprise, disgust, anger, neutral, sad, and fearful at 42.0%; chance 14.3%. Expression classification was stable across viewpoints. Representational similarity heatmaps indicated that image similarities within identities varied more by viewpoint than by expression. We conclude that an identity-trained, deep network retains shape-deformable information about expression and viewpoint, along with identity, in a unified form—consistent with a recent hypothesis for ventral visual stream processing.

Emerged human-like facial expression representation in a deep convolutional neural network

“…1C ). The results indicated that the expression-selective units spontaneously emerged in the VGG-Face pretrained for face identity recognition, which echoed previous findings ( 19 , 20 ).…”

“…It should be noted that, in our study, the classification accuracies of the expression-selective units in the pretrained VGG-Face were worse than the performance of expression recognition in humans, which was consistent with a recent finding showing that the identity-trained DCNN retained expression information but with expression recognition accuracies far below human performance ( 20 ). The reason for the decreased expression recognition performance deserves future investigation, although it is beyond the scope of the present study.…”

“…Thus, DCNNs could be a useful model simulating the processes of biological neural systems. More recently, several seminal studies have found that the DCNNs trained to recognize facial expression spontaneously developed facial identity recognition ability, and vice versa, suggesting that integrated representations of identity and expression may arise naturally within neural networks like humans do ( 19 , 20 ). However, a recent study found that face identity–selective units could spontaneously emerge in an untrained DCNN ( 21 ), which seemed to cast substantial doubt on the role of nurture in developing face perception and the abovementioned speculation.…”

What happens to our representation of identity as familiar faces age? Evidence from priming and identity aftereffects

“…PCA+LDA models suggest that there is enough physical commonality between images to support recognition of familiar faces across their lifespan, suggesting that multiple different representations are not necessary (Mileva et al, 2020). Likewise, DCNNs store diverse images of the same face (e.g., images that vary in viewpoint, lighting, expression) in the same region of face space (Colón et al, 2021;Hill et al, 2019;see O'Toole et al, 2018 for a summary), suggesting that images from multiple decades might reside in close proximity. Nevertheless, any theory derived from computational models should be complemented by behavioural data from humans.…”

“…Two lines of evidence show that DCNNs retain a representation of within-identity variability. First, visualization of the top layer of face space shows that images of an identity are clustered based on viewpoint, lighting, expression, and whether the input was a still image or video (Colón et al, 2021;Hill et al, 2019). Second, classification of image attributes (e.g., expression, viewpoint, whether the input was a still image or video) based on output at the top levels of DCNNs is highly accurate (Colón et al;Parde et al, 2017; see also Dhar et al, 2020 for expressivity as a measure of which image attributes are retained).…”