Modeling naturalistic face processing in humans with deep convolutional neural networks

Jiahui, Guo; Feilong, Ma; Castello, Matteo Visconti di Oleggio; Nastase, Samuel A.; Haxby, James V.; Gobbini, M. Ida

doi:10.32470/ccn.2022.1218-0

Cited by 21 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…the P100 (Luck et al, 1990), N170 (Bentin et al, 1996) and N400 components (Kutas & Federmeier, 2000), respectively (Figure 5a). Within neurotypicals, we found EEG representations peaking in similarity with the visual CNN at mid-layers (fourth and fifth; Jiahui et al, 2022) around mid-level temporal windows. Similarity with semantic computations also peaked around mid-latencies.…”

Section: Relationship With Electrophysiological Brain Componentsmentioning

confidence: 88%

“…Crucially, by associating electrophysiological signal and computational models, we found that the underlying neural computations of PS differed most with respect to the higher-level visual and semantic computations of deep neural networks models (DNNs). The late layers of visual DNNs have been previously linked to processing in human infero-temporal cortex (hIT; Güçlü & van Gerven, 2015;Jiahui et al, 2022.;Khaligh-Razavi & Kriegeskorte, 2014), peaking in the FFA (Khaligh-Razavi & Kriegeskorte, 2014), and functionally to higher-level visual feature representations such as parts of objects, whole objects and viewpoint invariant representations (Güçlü & van Gerven, 2015). These observations are consistent with the impaired whole face (Ramon et al, 2016) and feature representations (Caldara et al, 2005;Fiset et al, 2017) previously described in patient PS.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Neural computations in prosopagnosia

Faghel-Soubeyrand

Richoz

Waeber

et al. 2022

Preprint

View full text Add to dashboard Cite

We aimed to identify neural computations underlying the loss of face identification ability by modelling the brain activity of brain-lesioned patient PS, a well-documented case of acquired pure prosopagnosia. We collected a large dataset of high-density electrophysiological (EEG) recordings from PS and neurotypicals while they completed a one-back task on a stream of face, object, animal and scene images. We found reduced neural decoding of face identity around the N170 window in PS, and conjointly revealed normal non-face identification in this patient. We used Representational Similarity Analysis (RSA) to correlate human EEG representations with those of deep neural network (DNN) models of vision and caption-level semantics, offering a window into the neural computations at play in patient PS's deficits. Brain representational dissimilarity matrices (RDMs) were computed for each participant at 4 ms steps using cross-validated classifiers. PS's brain RDMs showed significant reliability across sessions, indicating meaningful measurements of brain representations with RSA even in the presence of significant lesions. Crucially, computational analyses were able to reveal PS's representational deficits in high-level visual and semantic brain computations. Such multi-modal data-driven characterisations of prosopagnosia highlight the complex nature of processes contributing to face recognition in the human brain.

show abstract

Section: Relationship With Electrophysiological Brain Componentsmentioning

confidence: 88%

Section: Discussionmentioning

confidence: 99%

Neural computations in prosopagnosia

Faghel-Soubeyrand

Richoz

Waeber

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…After obtaining these estimated maps, we calculated correlations between the target participant’s category-selective maps based on his/her own localizer data and the maps estimated from other participants’ data (hyperaligned or anatomically-aligned). We also calculated Cronbach’s alpha values (Feilong et al, 2018; Jiahui et al, 2020, 2022) across the multiple runs to measure the reliability of the category-selective maps for each participant and compared the correlations to the reliability values. To measure the local estimation performance and compare that to local reliabilities, we calculated correlations and Cronbach’s alphas in searchlights with a radius of 15 mm.…”

Section: Methodsmentioning

confidence: 99%

Cross-movie prediction of individualized functional topography

Jiahui

Feilong

Nastase

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Participant-specific, functionally-defined brain areas are usually mapped with functional localizers and estimated by making contrasts between responses to single categories of input. Naturalistic stimuli engage multiple brain systems in parallel, provide more ecologically plausible estimates of real-world statistics, and are friendly to special populations. The current study shows that cortical functional topographies in individual participants can be estimated with high fidelity from naturalistic stimuli. Importantly, we demonstrate that robust, individualized estimates can be obtained even when participants watched different movies, were scanned with different parameters/scanners, and were sampled from different institutes across the world. Our results create a foundation for future studies that allow researchers to estimate a broad range of functional topographies based on naturalistic movies and a normative database, making it possible to integrate high-level cognitive functions across datasets from laboratories worldwide.

show abstract

“…We trained our VAE models using the TensorFlow DisentanglementLib package [40]. To identify the best disentangled model for our fMRI analyses, we performed a hyperparameter search over model architectures (including beta-VAE [14] and FactorVAE [17]), number of latent dimensions (24, 32, 48, and 64), and architecture-specific disentanglement parameters (beta-VAE β ϵ [1,2,4,6,8,16], FactorVAE γ ϵ [10,20,30,40,50,100]). For every hyperparameter combination, we performed 10 random initializations.…”

Section: M1 Neural Net Architecture and Trainingmentioning

confidence: 99%

“…Recently, deep convolutional neural networks (DCNNs) trained on face recognition have been shown to learn effective face representations that provide a good match to human behavior [8], but such discriminatively trained models are difficult to interpret [9] and provide a poor match to human neural data [10]. Alternatively, deep generative models have been shown to provide a good match to human fMRI face processing data [11].…”

Section: Introductionmentioning

confidence: 99%

Disentangled deep generative models reveal coding principles of the human face processing network

Soulos

Işık

2023

Preprint

View full text Add to dashboard Cite

Despite decades of research, much is still unknown about the computations carried out in the human face processing network. Recently deep networks have been proposed as a computational account of human visual processing, but while they provide a good match to neural data throughout visual cortex, they lack interpretability. We introduce a method for interpreting brain activity using a new class of deep generative models, disentangled representation learning models, which learn a low-dimensional latent space that "disentangles" different semantically meaningful dimensions of faces, such as rotation, lighting, or hairstyle, in an unsupervised manner by enforcing statistical independence between dimensions. We find that the majority of our model's learned latent dimensions are interpretable by human raters. Further, these latent dimensions serve as a good encoding model for human fMRI data. We next investigated the representation of different latent dimensions across face-selective voxels. We find a gradient from low- to high-level face feature representations along posterior to anterior face-selective regions, corroborating prior models of human face recognition. Interestingly, though, we find no spatial segregation between identity-relevant and irrelevant face features. Finally, we provide new insight into the few "entangled" (uninterpretable) dimensions in our model by showing that they match responses across the ventral stream and carry significant information about facial identity. Disentangled face encoding models provide an exciting alternative to standard "black box" deep learning approaches for modeling and interpreting human brain data.

show abstract

Modeling naturalistic face processing in humans with deep convolutional neural networks

Cited by 21 publications

References 0 publications

Neural computations in prosopagnosia

Neural computations in prosopagnosia

Cross-movie prediction of individualized functional topography

Disentangled deep generative models reveal coding principles of the human face processing network

Contact Info

Product

Resources

About