Face and Image Representation in Deep CNN Features

Parde, Connor J.; Castillo, Carlos D.; Hill, Matthew Q.; Colón, Yvette; Sankaranarayanan, Swami; Chen, Jun-Cheng; O’Toole, Alice J.

doi:10.1109/fg.2017.85

Cited by 25 publications

(44 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, we demonstrate that information related to social and personality traits is retained in the top‐level features of a DCNN trained for face identification. This finding complements previous work illustrating that DCNNs retain specific information about images that is not directly relevant for object/face recognition (Hong, Yamins, Majaj, & Dicarlo, ; Parde et al., ). Second, we show that DCNN face “identity” representations predict human‐generated social‐trait inferences, both at the individual‐trait level and at the level of full trait profiles.…”

Section: Discussionsupporting

confidence: 89%

“…Further, a recent study by Todorov and Porter () showed that image characteristics, which include photometric variables such as illumination and image quality, are linked to social‐trait perception. It has also been shown that DCNNs trained for face identification retain information pertaining to these image characteristics (Parde et al., ). Together, these studies provide further evidence that identity and social traits can coexist in a unitary representation.…”

Section: Discussionmentioning

confidence: 99%

“…The topic of why DCNNs generalize is discussed in depth in O'Toole, Castillo, Parde, Hill, and Chellappa () and C. Zhang, Bengio, Hardt, Recht, and Vinyals (). Presently, the surprising generalizability of DCNNs implies that these networks have the potential to model trait inferences based on expression and image‐characteristics in addition to facial structure (cf., O'Toole et al., ; Parde et al., ). Future work should consider ways to dissect the sources of face‐identity and face‐image information that allow DCNNs to predict social trait inferences (see methods proposed by Yosinski, Clune, Nguyen, Fuchs, & Lipson, ).…”

Section: Discussionmentioning

confidence: 99%

See 2 more Smart Citations

Social Trait Information in Deep Convolutional Neural Networks Trained for Face Identification

Parde

Castillo

et al. 2019

Cognitive Science

Self Cite

View full text Add to dashboard Cite

Faces provide information about a person's identity, as well as their sex, age, and ethnicity. People also infer social and personality traits from the face — judgments that can have important societal and personal consequences. In recent years, deep convolutional neural networks (DCNNs) have proven adept at representing the identity of a face from images that vary widely in viewpoint, illumination, expression, and appearance. These algorithms are modeled on the primate visual cortex and consist of multiple processing layers of simulated neurons. Here, we examined whether a DCNN trained for face identification also retains a representation of the information in faces that supports social‐trait inferences. Participants rated male and female faces on a diverse set of 18 personality traits. Linear classifiers were trained with cross validation to predict human‐assigned trait ratings from the 512 dimensional representations of faces that emerged at the top‐layer of a DCNN trained for face identification. The network was trained with 494,414 images of 10,575 identities and consisted of seven layers and 19.8 million parameters. The top‐level DCNN features produced by the network predicted the human‐assigned social trait profiles with good accuracy. Human‐assigned ratings for the individual traits were also predicted accurately. We conclude that the face representations that emerge from DCNNs retain facial information that goes beyond the strict limits of their training.

show abstract

Section: Discussionsupporting

confidence: 89%

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Social Trait Information in Deep Convolutional Neural Networks Trained for Face Identification

Parde

Castillo

et al. 2019

Cognitive Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…These layers first expand the representation, and then condense it into a compact face descriptor. The representation that emerges from a DCNN trained for face identity retains both invariant (identity, gender) and changeable (viewpoint, illumination) aspects of faces ( Hill et al, 2019 ; O’Toole et al, 2018 ; Parde et al, 2017 ).…”

Section: Introductionmentioning

confidence: 99%

Facial expression is retained in deep networks trained for face identification

2021

View full text Add to dashboard Cite

Facial expressions distort visual cues for identification in two-dimensional images. Face processing systems in the brain must decouple image-based information from multiple sources to operate in the social world. Deep convolutional neural networks (DCNN) trained for face identification retain identity-irrelevant, image-based information (e.g., viewpoint). We asked whether a DCNN trained for identity also retains expression information that generalizes over viewpoint change. DCNN representations were generated for a controlled dataset containing images of 70 actors posing 7 facial expressions (happy, sad, angry, surprised, fearful, disgusted, neutral), from 5 viewpoints (frontal, 90° and 45° left and right profiles). Two-dimensional visualizations of the DCNN representations revealed hierarchical groupings by identity, followed by viewpoint, and then by facial expression. Linear discriminant analysis of full-dimensional representations predicted expressions accurately, mean 76.8% correct for happiness, followed by surprise, disgust, anger, neutral, sad, and fearful at 42.0%; chance 14.3%. Expression classification was stable across viewpoints. Representational similarity heatmaps indicated that image similarities within identities varied more by viewpoint than by expression. We conclude that an identity-trained, deep network retains shape-deformable information about expression and viewpoint, along with identity, in a unified form—consistent with a recent hypothesis for ventral visual stream processing.

show abstract

“…In addition, a little research has explored the implicit feature representations learned by DNNs that are associated with high-level perception. The studies of McCurrie et al (2018) and Parde et al (2019) are examples in which implicit feature representations learned by DNNs are used to better understand subjective social traits of face perception.…”

Section: Introductionmentioning

confidence: 99%

Putative Ratios of Facial Attractiveness in a Deep Neural Network

Song¹,

Liang²,

Kumada³

et al. 2020

Preprint

View full text Add to dashboard Cite

Empirical evidence has shown that there is an ideal arrangement of facial features (ideal ratios) that can optimize the attractiveness of a person's face. These putative ratios define facial attractiveness in terms of spatial relations and provide important rules for measuring the attractiveness of a face. In this paper, we show that a deep neural network (DNN) model can learn putative ratios based only on categorical annotation when no annotated facial features for attractiveness are explicitly given. To this end, we conducted three experiments. In Experiment 1, we trained a DNN model to recognize facial attractiveness using four category-specific neurons (female/male $\times$ high/low attractiveness). In Experiment 2, face-like images were generated by reversing the DNN model (e.g., deconvolution). These images depict the intuitive attributes of the four categories of facial attractiveness and reveal certain consistencies with reported evidence on the putative ratios of facial attractiveness. In Experiment 3, simulated psychophysical experiments on facial images with varying ratios of features reveal changes in the activity of the category-specific neurons that are remarkably similar to those of human judgements reported in a previous study. These results show that the trained DNN model can learn putative ratios as key features for the representation of facial attractiveness. These findings advance our understanding of facial attractiveness and high-level human perception.

show abstract

Face and Image Representation in Deep CNN Features

Cited by 25 publications

References 12 publications

Social Trait Information in Deep Convolutional Neural Networks Trained for Face Identification

Social Trait Information in Deep Convolutional Neural Networks Trained for Face Identification

Facial expression is retained in deep networks trained for face identification

Putative Ratios of Facial Attractiveness in a Deep Neural Network

Contact Info

Product

Resources

About