Deep learning algorithms reveal a new visual-semantic representation of familiar faces in human perception and memory

Shoham, Adva; Grosbard, Idan; Patashnik, Or; Cohen–Or, Daniel; Yovel, Galit

doi:10.1101/2022.10.16.512398

Cited by 17 publications

(3 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Training DNNs initially on blurred images also provided insights into the potential advantage of the initial low acuity of infants' vision (Vogelsang et al, 2018). Such and many other modifications (e.g., multi-modal self-supervised image-language training, in the way DNNs are built and trained may generate perceptual effects that are more human-like (Shoham, Grosbard, Patashnik, Cohen-Or, & Yovel, 2022). Yet even current DNNs can advance our understanding of the nature of the high-level representations that are required for face and object recognition (Abudarham, Grosbard, & Yovel, 2021;Hill et al, 2019), which are still undefined in current neural and cognitive models.…”

Section: Introductionmentioning

confidence: 99%

Deep problems with neural network models of human vision

Bowers¹,

Dujmović²,

Montero³

et al. 2022

Behav Brain Sci

View full text Add to dashboard Cite

Deep neural networks (DNNs) have had extraordinary successes in classifying photographic images of objects and are often described as the best models of biological vision. This conclusion is largely based on three sets of findings: (1) DNNs are more accurate than any other model in classifying images taken from various datasets, (2) DNNs do the best job in predicting the pattern of human errors in classifying objects taken from various behavioral datasets, and (3) DNNs do the best job in predicting brain signals in response to images taken from various brain datasets (e.g., single cell responses or fMRI data). However, these behavioral and brain datasets do not test hypotheses regarding what features are contributing to good predictions and we show that the predictions may be mediated by DNNs that share little overlap with biological vision. More problematically, we show that DNNs account for almost no results from psychological research. This contradicts the common claim that DNNs are good, let alone the best, models of human object recognition. We argue that theorists interested in developing biologically plausible models of human vision need to direct their attention to explaining psychological findings. More generally, theorists need to build models that explain the results of experiments that manipulate independent variables designed to test hypotheses rather than compete on making the best predictions. We conclude by briefly summarizing various promising modelling approaches that focus on psychological data.

show abstract

Section: Introductionmentioning

confidence: 99%

Deep problems with neural network models of human vision

Bowers¹,

Dujmović²,

Montero³

et al. 2022

Behav Brain Sci

View full text Add to dashboard Cite

show abstract

“…Further, object-trained networks are currently the best model of face-specific neural responses in the primate brain ( 16 , 17 ) and even appear to contain units selectively responsive to faces ( 18 , 19 ). A third possibility is that none of the above training regimes might be able to capture all classic signatures of human face perception, and something else might be required, such as a face-specific inductive bias ( 20 , 21 ) or a higher-level semantic processing of faces ( 22 ), to capture human behavioral signatures of face processing. Finally, these hypotheses are not mutually exclusive, and it is possible that different signatures of human face processing may result from optimization for different tasks.…”

mentioning

confidence: 99%

Behavioral signatures of face perception emerge in deep neural networks optimized for face recognition

Dobs

Yuan

Martinez

et al. 2023

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Human face recognition is highly accurate and exhibits a number of distinctive and well-documented behavioral “signatures” such as the use of a characteristic representational space, the disproportionate performance cost when stimuli are presented upside down, and the drop in accuracy for faces from races the participant is less familiar with. These and other phenomena have long been taken as evidence that face recognition is “special”. But why does human face perception exhibit these properties in the first place? Here, we use deep convolutional neural networks (CNNs) to test the hypothesis that all of these signatures of human face perception result from optimization for the task of face recognition. Indeed, as predicted by this hypothesis, these phenomena are all found in CNNs trained on face recognition, but not in CNNs trained on object recognition, even when additionally trained to detect faces while matching the amount of face experience. To test whether these signatures are in principle specific to faces, we optimized a CNN on car discrimination and tested it on upright and inverted car images. As we found for face perception, the car-trained network showed a drop in performance for inverted vs. upright cars. Similarly, CNNs trained on inverted faces produced an inverted face inversion effect. These findings show that the behavioral signatures of human face perception reflect and are well explained as the result of optimization for the task of face recognition, and that the nature of the computations underlying this task may not be so special after all.

show abstract

“…Training DNNs initially on blurred images also provided insights into the potential advantage of the initial low acuity of infants’ vision (Vogelsang et al, 2018). Such and many other modifications (e.g., multi-modal self-supervised image-language training, Radford et al, 2021) in the way DNNs are built and trained may generate perceptual effects that are more human-like (Shoham, Grosbard, Patashnik, Cohen-Or, & Yovel, 2022). Yet even current DNNs can advance our understanding of the nature of the high-level representations that are required for face and object recognition (Abudarham, Grosbard, & Yovel, 2021; Hill et al, 2019), which are still undefined in current neural and cognitive models.…”

mentioning

confidence: 99%

Why psychologists should embrace rather than abandon DNNs

Yovel,

Abudarham

2023

Behav Brain Sci

View full text Add to dashboard Cite

Deep neural networks (DNNs) are powerful computational models, which generate complex, high-level representations that were missing in previous models of human cognition. By studying these high-level representations, psychologists can now gain new insights into the nature and origin of human high-level vision, which was not possible with traditional handcrafted models. Abandoning DNNs would be a huge oversight for psychological sciences.

show abstract

Deep learning algorithms reveal a new visual-semantic representation of familiar faces in human perception and memory

Cited by 17 publications

References 43 publications

Deep problems with neural network models of human vision

Deep problems with neural network models of human vision

Behavioral signatures of face perception emerge in deep neural networks optimized for face recognition

Why psychologists should embrace rather than abandon DNNs

Contact Info

Product

Resources

About