Capturing human categorization of natural images by combining deep networks and cognitive models

Abstract

Human categorization is one of the most important and successful targets of cognitive modeling, with decades of model development and assessment using simple, low-dimensional artificial stimuli. However, it remains unclear how these findings relate to categorization in more natural settings, involving complex, high-dimensional stimuli. Here, we take a step towards addressing this question by modeling human categorization over a large behavioral dataset, comprising more than 500,000 judgments over 10,000 natural images from ten object categories. We apply a range of machine learning methods to generate candidate representations for these images, and show that combining rich image representations with flexible cognitive models captures human decisions best. We also find that in the high-dimensional representational spaces these methods generate, simple prototype models can perform comparably to the more complex memory-based exemplar models dominant in laboratory settings.

The Geometry of Concept Learning

“…8), shedding light on a 40-year-old debate 5 . These predictions are consistent with a recent demonstration that a prototype-based rule can match the performance of an exemplar model on categorization of familiar high dimensional stimuli 58 . We go beyond prior work by (1) demonstrating that prototype learning achieves superior performance on fewshot learning of novel naturalistic concepts, (2) precisely characterizing the tradeoff as a joint function of concept manifold dimensionality and the number of training examples (Fig.…”

“…Such increased model complexity raises foundational questions about the appropriate comparisons between brains and machine based models 55 . Previous approaches based on behavioral performance 16,25,[56][57][58] , neuron 46 or circuit 49 matching, linear regression between representations 22 , or representational similarity analysis 24 , reveal a reasonable match between the two. However, our higher-resolution decomposition of performance into a fundamental set of observable geometric properties reveals significant mismatches ( Fig.…”

From convolutional neural networks to models of higher‐level cognition (and back again)

“…CNNs can be used to supply representations for more complex naturalistic images, which can be further modified to better reflect human judgments before being input into the same kinds of cognitive model (middle pathways). 50,64,73 End-to-end models offer the opportunity to solve both of these problems simultaneously and learn a representation for naturalistic stimuli that satisfies the constraints inherent in higher-level cognitive models (right pathway). 136 Many other components of the modern deep learning framework arose during this period from the collaboration of psychologists, neuroscientists, and the computational vision community, including the development of hierarchical feed-forward visual models based on stacks of nonlinear feature maps and pooling between layers.…”

“…This acts to sharpen or lessen the influence of exemplars on subsequent category judgments, and therefore allow the model to controls for overall stimulus discriminability in the relevant psychological space. 94 Battleday et al 50 found that CNNs provided the best representational basis for modeling the human categorization judgments, outperforming deep unsupervised and traditional computer vision methods. Indeed, the choice of stimulus representation affected overall performance to a much greater extent than the choice of categorization model.…”

“…There are known cases in which prototype and exemplar models make similar predictions, for example, if category representations are wellcaptured by simple Gaussians, but how these situations relate to high-dimensional representations of complex stimuli remains unclear. Battleday et al 50 conducted a simulation study that investigated how the number of dimensions and training samples affected the different modeling strategies. They found that while indeed in low-dimensional spaces exemplar models outperform prototype models, no such difference exists in high-dimensional representational spaces after training with very large numbers of stimuli.…”