Evolving Images for Visual Neurons Using a Deep Generative Network Reveals Coding Principles and Neuronal Preferences

Ponce, Carlos R.; Xiao, Will; Schade, Peter F.; Hartmann, Till S.; Kreiman, Gabriel; Livingstone, Margaret S.

doi:10.1016/j.cell.2019.04.005

Cited by 203 publications

(210 citation statements)

References 44 publications

Supporting

Mentioning

199

Contrasting

Order By: Relevance

“…These results suggest that a warm bias is a consistent feature of IT cells. The warm bias we report is also consistent with the colors of optimal stimuli obtained in neural-response-guided machine learning experiments of IT cells (Ponce et al, 2019).…”

Section: Discussionsupporting

confidence: 87%

Color tuning of neurons in face patches of macaque inferior temporal cortex

Duyck

Gruen

Tello

et al. 2019

Preprint

View full text Add to dashboard Cite

How are face-specific color signals encoded by the brain? We addressed this question by measuring color responses of face-selective cells in alert macaque monkey, using fMRI-guided microelectrode recording of the middle and anterior face patches. Many face-selective neurons showed broad color tuning when assessed using images that preserved the luminance contrast relationships of the original face photographs. A Fourier analysis of the color-tuning responses uncovered two components. The first harmonic showed a bias towards the L>M pole of the L-M cardinal axis of cone-opponent color space (appearing reddish), which suggests that face cells encode a prior about the color component of faces that is important for social signaling (blood perfusion). The second harmonic showed a bias for colors that modulate the S-cone cardinal axis, which may relate to the computation of animacy by IT cells. Taken together, these results uncover a putative physiological basis for the role of color in face perception and show that chromatic signatures corresponding to the cardinal chromatic mechanisms are evident not only in subcortical circuits, as previously known, but also far along the visualprocessing hierarchy, within inferior temporal cortex.Significance: It is not known how the brain processes combined color and face information. The present results fill this gap in knowledge by uncovering the color tuning properties of face-selective neurons. The results further our understanding of the neural mechanisms that make color an informative cue for social communication.

show abstract

Section: Discussionsupporting

confidence: 87%

Color tuning of neurons in face patches of macaque inferior temporal cortex

Duyck

Gruen

Tello

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, deep neural networks (DNN) using feedforward hierarchies of convolutional features to process images have reached and even surpassed human category-level recognition performance (He et al, 2016;Kietzmann et al, 2018;Lindsay, 2020;Russakovsky et al, 2015;Yamins & DiCarlo, 2016). Despite being developed as computer vision tools, DNNs trained to recognise objects in images are also unsurpassed at predicting how natural images are represented in high-level ventral visual areas of the human and non-human primate brain (Agrawal et al, 2014;Bashivan et al, 2019;Cadieu et al, 2014;Cichy et al, 2016;Devereux et al, 2018;Eickenberg et al, 2017;Güçlü & van Gerven, 2015;Horikawa & Kamitani, 2017;Kubilius et al, 2018;Lindsay, 2020;Ponce et al, 2019;Schrimpf et al, 2018;Xu & Vaziri-Pashkam, 2020;Yamins & DiCarlo, 2016). There is some variability in the accuracy with which different recent DNNs can predict high-level visual representations Xu & Vaziri-Pashkam, 2020;Zeman et al, 2020), despite broadly high performance.…”

Section: Introductionmentioning

confidence: 99%

“…Many studies reporting high performance of DNNs as models of visual cortex allow linear reweighting of individual features (e.g. Agrawal et al, 2014;Bashivan et al, 2019;Cadieu et al, 2014;Güçlü & van Gerven, 2015;Horikawa & Kamitani, 2017;Ponce et al, 2019;, while others treat the representations within a layer of a network as fixed (e.g. Truzzi & Cusack, 2020;Zeman et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

Diverse deep neural networks all predict human IT well, after training and fitting

Storrs

Kietzmann

Walther

et al. 2020

Preprint

View full text Add to dashboard Cite

Deep neural networks (DNNs) trained on object recognition provide the best current models of high-level visual areas in the brain. What remains unclear is how strongly network design choices, such as architecture, task training, and subsequent fitting to brain data contribute to the observed similarities. Here we compare a diverse set of nine DNN architectures on their ability to explain the representational geometry of 62 isolated object images in human inferior temporal (hIT) cortex, as measured with functional magnetic resonance imaging. We compare untrained networks to their task-trained counterparts, and assess the effect of fitting them to hIT using a cross-validation procedure. To best explain hIT, we fit a weighted combination of the principal components of the features within each layer, and subsequently a weighted combination of layers. We test all models across all stages of training and fitting for their correlation with the hIT representational dissimilarity matrix (RDM) using an independent set of images and subjects. We find that trained models significantly outperform untrained models (accounting for 57% more of the explainable variance), suggesting that features representing natural images are important for explaining hIT.Model fitting further improves the alignment of DNN and hIT representations (by 124%), suggesting that the relative prevalence of different features in hIT does not readily emerge from the particular ImageNet object-recognition task used to train the networks.Finally, all DNN architectures tested achieved equivalent high performance once trained and fitted. Similar ability to explain hIT representations appears to be shared among deep feedforward hierarchies of nonlinear features with spatially restricted receptive fields.

show abstract

“…Here, we used GANs to coordinate high-dimensional image manipulations and constrain them to remain approximately natural. Such coordinated high-dimensional stimulation may be particularly useful for characterizing multi-dimensional stimulus representation in physiological experiments [23]. It is well known that neural responses to natural input can be quite different from responses to synthetic stimuli [48] and generative image models such as GANs can help constrain stimulation to remain approximately natural.…”

Section: Discussionmentioning

confidence: 99%

“…GANs have recently been successful at generating extremely realistic looking images (see for example, [21,22]). Furthermore, GANs seem to recover a parametrization of the manifold of natural images that matches aspects of human perception [20] and predicts neural responses in non-human primates [23]. By expressing manipulations of naturalistic images along this approximation of the manifold of natural images, we can characterize complex invariances in human visual perception without a need to rely on analogies between human processing and processing in deep artificial neural network algorithms.…”

mentioning

confidence: 99%

Contrast invariant tuning in human perception of image content

Fruend

Patel

Stalker

2019

Preprint

View full text Add to dashboard Cite

Higher levels of visual processing are progressively more invariant to low-level visual factors such as contrast. Although this invariance trend has been well documented for simple stimuli like gratings and lines, it is difficult to characterize such invariances in images with naturalistic complexity. Here, we use a generative image model based on a hierarchy of learned visual features-a Generative Adversarial Networkto constrain image manipulations to remain within the vicinity of the manifold of natural images. This allows us to quantitatively characterize visual discrimination behaviour for naturalistically complex, non-linear image manipulations. We find that human tuning to such manipulations has a factorial structure. The first factor governs image contrast with discrimination thresholds following a power law with an exponent between 0.5 and 0.6, similar to contrast discrimination performance for simpler stimuli. A second factor governs image content with approximately constant discrimination thresholds throughout the range of images studied. These results support the idea that human perception factors out image contrast relatively early on, allowing later stages of processing to extract higher level image features in a stable and robust way.

show abstract

Evolving Images for Visual Neurons Using a Deep Generative Network Reveals Coding Principles and Neuronal Preferences

Cited by 203 publications

References 44 publications

Color tuning of neurons in face patches of macaque inferior temporal cortex

Color tuning of neurons in face patches of macaque inferior temporal cortex

Diverse deep neural networks all predict human IT well, after training and fitting

Contrast invariant tuning in human perception of image content

Contact Info

Product

Resources

About