Revealing the multidimensional mental representations of natural objects underlying human similarity judgments

Hebart, Martin N.; Zheng, Charles; Pereira, Francisco C.; Baker, Chris I.

doi:10.31234/osf.io/7wrgh

Cited by 15 publications

(20 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Perception of similarities is the outcome of an active comparison process between multiple objects. For complex real-world objects, it relies on the integration of information across multiple feature dimensions, and this integration is characterized by a high degree of flexibility (Medin et al, 1993;Hebart et al, 2020). As such, it is perhaps not surprising that reported perceived similarities between objects vary across observers (see also Goldstone, 1994;Goldstone & Son, 2012), and have behavioural relevance in performance of object discrimination, as shown in the current study.…”

Section: Discussionsupporting

confidence: 59%

Activity in perirhinal and entorhinal cortex predicts perceived visual similarities among category exemplars with highest precision

Blumenthal

Cb³

et al. 2021

Preprint

View full text Add to dashboard Cite

Observers perceive their visual environment in unique ways. How ventral visual stream (VVS) regions represent subjectively perceived object characteristics remains poorly understood. We hypothesized that the visual similarity between objects that observers perceive is reflected with highest fidelity in neural activity patterns in perirhinal and anterolateral entorhinal cortex at the apex of the VVS object-processing hierarchy. To address this issue with fMRI, we administered a task that required discrimination between images of exemplars from real-world categories. Further, we obtained ratings of perceived visual similarities. We found that perceived visual similarities predicted discrimination performance in an observer-specific manner. As anticipated, activity patterns in perirhinal and anterolateral entorhinal cortex predicted perceived similarity structure, including those aspects that are observer-specific, with higher fidelity than any other region examined. Our findings provide new evidence that representations of the visual world at the apex of the VVS differ across observers in ways that influence behaviour.

show abstract

Section: Discussionsupporting

confidence: 59%

Activity in perirhinal and entorhinal cortex predicts perceived visual similarities among category exemplars with highest precision

Blumenthal

Cb³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Here, we presented a first foray into this domain by showing that ecoset training leads to an increase in face-selective units in final network layers. Moving further into the domain of behavior, it will be of interest to perform in-depth tests of ecoset-trained networks (supervised or unsupervised) to compare their task performance and error distributions against human behavioral data (39)(40)(41)(42).…”

Section: Discussionmentioning

confidence: 99%

An ecologically motivated image dataset for deep learning yields better models of human vision

Mehrer

Spoerer

Jones

et al. 2021

Proc. Natl. Acad. Sci. U.S.A.

121

109

View full text Add to dashboard Cite

Deep neural networks provide the current best models of visual information processing in the primate brain. Drawing on work from computer vision, the most commonly used networks are pretrained on data from the ImageNet Large Scale Visual Recognition Challenge. This dataset comprises images from 1,000 categories, selected to provide a challenging testbed for automated visual object recognition systems. Moving beyond this common practice, we here introduce ecoset, a collection of >1.5 million images from 565 basic-level categories selected to better capture the distribution of objects relevant to humans. Ecoset categories were chosen to be both frequent in linguistic usage and concrete, thereby mirroring important physical objects in the world. We test the effects of training on this ecologically more valid dataset using multiple instances of two neural network architectures: AlexNet and vNet, a novel architecture designed to mimic the progressive increase in receptive field sizes along the human ventral stream. We show that training on ecoset leads to significant improvements in predicting representations in human higher-level visual cortex and perceptual judgments, surpassing the previous state of the art. Significant and highly consistent benefits are demonstrated for both architectures on two separate functional magnetic resonance imaging (fMRI) datasets and behavioral data, jointly covering responses to 1,292 visual stimuli from a wide variety of object categories. These results suggest that computational visual neuroscience may take better advantage of the deep learning framework by using image sets that reflect the human perceptual and cognitive experience. Ecoset and trained network models are openly available to the research community.

show abstract

“…One of the central goals of computational visual neuroscience is a model that can predict neural representations in visual cortex at multiple levels of granularity, from single neuron responses to the aggregated population signals measured via fMRI, and can also predict the perceptual properties of our visual systems, as measured in behavioural experiments (Funke et al, 2020;Hebart et al, 2020;Rajalingham et al, 2018;Schrimpf et al, 2018;. We were therefore interested in whether models that predicted fMRI-based representational dissimilarities in human IT better also predicted electrophysiological or behavioural object perception data better.…”

Section: Differences In Hit Correspondence Between Trained Models Do mentioning

confidence: 99%

Diverse deep neural networks all predict human IT well, after training and fitting

Storrs

Kietzmann

Walther

et al. 2020

Preprint

View full text Add to dashboard Cite

Deep neural networks (DNNs) trained on object recognition provide the best current models of high-level visual areas in the brain. What remains unclear is how strongly network design choices, such as architecture, task training, and subsequent fitting to brain data contribute to the observed similarities. Here we compare a diverse set of nine DNN architectures on their ability to explain the representational geometry of 62 isolated object images in human inferior temporal (hIT) cortex, as measured with functional magnetic resonance imaging. We compare untrained networks to their task-trained counterparts, and assess the effect of fitting them to hIT using a cross-validation procedure. To best explain hIT, we fit a weighted combination of the principal components of the features within each layer, and subsequently a weighted combination of layers. We test all models across all stages of training and fitting for their correlation with the hIT representational dissimilarity matrix (RDM) using an independent set of images and subjects. We find that trained models significantly outperform untrained models (accounting for 57% more of the explainable variance), suggesting that features representing natural images are important for explaining hIT.Model fitting further improves the alignment of DNN and hIT representations (by 124%), suggesting that the relative prevalence of different features in hIT does not readily emerge from the particular ImageNet object-recognition task used to train the networks.Finally, all DNN architectures tested achieved equivalent high performance once trained and fitted. Similar ability to explain hIT representations appears to be shared among deep feedforward hierarchies of nonlinear features with spatially restricted receptive fields.

show abstract

Revealing the multidimensional mental representations of natural objects underlying human similarity judgments

Cited by 15 publications

References 42 publications

Activity in perirhinal and entorhinal cortex predicts perceived visual similarities among category exemplars with highest precision

Activity in perirhinal and entorhinal cortex predicts perceived visual similarities among category exemplars with highest precision

An ecologically motivated image dataset for deep learning yields better models of human vision

Diverse deep neural networks all predict human IT well, after training and fitting

Contact Info

Product

Resources

About