Spatiotemporal dynamics of similarity-based neural representations of facial identity

Abstract

Humans' remarkable ability to quickly and accurately discriminate among thousands of highly similar complex objects demands rapid and precise neural computations. To elucidate the process by which this is achieved, we used magnetoencephalography to measure spatiotemporal patterns of neural activity with high temporal resolution during visual discrimination among a large and carefully controlled set of faces. We also compared these neural data to lower level "image-based" and higher level "identitybased" model-based representations of our stimuli and to behavioral similarity judgments of our stimuli. Between ∼50 and 400 ms after stimulus onset, face-selective sources in right lateral occipital cortex and right fusiform gyrus and sources in a control region (left V1) yielded successful classification of facial identity. In all regions, early responses were more similar to the image-based representation than to the identity-based representation. In the face-selective regions only, responses were more similar to the identity-based representation at several time points after 200 ms. Behavioral responses were more similar to the identity-based representation than to the image-based representation, and their structure was predicted by responses in the face-selective regions. These results provide a temporally precise description of the transformation from low-to high-level representations of facial identity in human face-selective cortex and demonstrate that faceselective cortical regions represent multiple distinct types of information about face identity at different times over the first 500 ms after stimulus onset. These results have important implications for understanding the rapid emergence of fine-grained, highlevel representations of object identity, a computation essential to human visual expertise.face processing | magnetoencephalography | decoding | representational similarity analysis | face identity H umans can discriminate among thousands of highly similar and complex visual patterns, such as face identity, in less than half a second (1, 2). Efficient within-category discrimination of facial identity is important for real-world decisions (e.g., classifying a person as a friend or stranger) and social interactions. Progress has been made in elucidating the neural mechanisms underlying the discrimination of individual face identities in humans. Using fMRI, these studies demonstrate that individual face identities are represented by spatially distributed patterns of neural activity within occipitotemporal cortex (3-13). Because of the poor temporal resolution in fMRI studies (typically around 2 s), however, our understanding of the neural basis of discrimination among complex visual patterns in humans remains limited. For example, within a given region, different information relevant to discrimination may be represented at different times over the first few 100 ms after stimulus onset. However, current models of the neural basis of face recognition in humans do not typically allow for this possibility, becau...

Spatio-temporal dynamics of face perception

“…In ROI analyses we similarly found an early peak in fMRI-EEG correlations in V1, and a later peak in OFA and FFA. Our time trajectories are similar to those reported in a previous MEG study (Vida et al, 2017), which found first peaks for V1, OFA and FFA at around 150 ms and later peaks ~250 ms. Both in our and their results, information in V1 clearly dominated early in time, and later in time, the information from OFA and FFA was relatively more prominent, though mainly not surpassing correlations with V1.…”

“…In left OFA, the correlations remained significant until 480 ms. The overall spatio-temporal structure found here was quite similar to that reported earlier in a MEG study (Vida et al, 2017) looking at the coding of face identities in the brain, finding accurate classifications in the left V1 around 150 ms, and relatively higher decoding accuracy for the right LO and the right FG around 250 ms. To ensure that our results were not due to selecting the five most informative voxels, the ROI analyses were repeated using all the voxels within a given BALSA-area. The results were similar except that the right FFA did not correlate significantly with EEG ( Supplementary Figure 3).…”

“…Coding of identities is usually found later. Although earliest results are found from ~100 ms onwards (Ambrus, Kaiser, Cichy, & Kovács, 2019;Vida, Nestor, Plaut, & Behrmann, 2017), higher-level identity representations surpassing low-level visual information (Vida et al, 2017), as well as sensitivity to familiar faces (Schweinberger & Neumann, 2016), are found later, at around 250 ms, and differentiation of same sex identities only after 400 ms (Ambrus et al, 2019).…”

Spatiotemporal dynamics in human visual cortex rapidly encode the emotional content of faces

“…Fast decoding of object category was achieved at 100 ms from small neuronal populations in primates (Hung & Poggio, 2005) and from invasively recorded responses in human visual cortex (Li & Lu, 2009). Furthermore, recent applications of MVPA to electrophysiological data have resolved face identity processing to early latencies (50-70 ms after stimulus onset; Davidesco et al, 2014;Nemrodov et al, 2016;Vida, Nestor, Plaut, & Behrmann, 2017). In addition to revealing the temporal dynamics of visual processing, multivariate methods have furthered our understanding of the transformations performed by cells in macaque face patches to encode face identity (Chang & Tsao, 2017) and have allowed face reconstruction based on non-invasive neural data in humans (Nemrodov et al, 2018;Nestor, Plaut, & Behrmann, 2016).…”

Temporal Dynamics of the Neural Representation of Social Relationships

“…Six dissimilarity matrix (DMs) models representing three types of information were used to this end: visual properties, character identity, and social relationships (Figure 2). All models were compared to neural distance across two different facial expressions within each character identity (Vida et al, 2016). The primary model of visual properties was the Euclidean distance between responses of the C2 layer of HMAX, which simulates the complex visual cell response to an input image (Riesenhuber and Poggio, 1999).…”