Can machine learning account for human visual object shape similarity judgments?

German, Joseph; Jacobs, Robert A.

doi:10.1016/j.visres.2019.12.001

Cited by 13 publications

(8 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, similarity judgments break down for unfamiliar objects. For example, German and Jacobs (2020) measured human similarity judgments between pairs of novel part-based naturalistic objects (fribbles) presented across multiple viewpoints. These judgments were then compared with the similarities observed in DNNs in response to the same stimuli.…”

Section: The Empirical Problem With Claiming Dnns and Human Vision Ar...mentioning

confidence: 99%

Deep problems with neural network models of human vision

Bowers¹,

Dujmović²,

Montero³

et al. 2022

Behav Brain Sci

View full text Add to dashboard Cite

Deep neural networks (DNNs) have had extraordinary successes in classifying photographic images of objects and are often described as the best models of biological vision. This conclusion is largely based on three sets of findings: (1) DNNs are more accurate than any other model in classifying images taken from various datasets, (2) DNNs do the best job in predicting the pattern of human errors in classifying objects taken from various behavioral datasets, and (3) DNNs do the best job in predicting brain signals in response to images taken from various brain datasets (e.g., single cell responses or fMRI data). However, these behavioral and brain datasets do not test hypotheses regarding what features are contributing to good predictions and we show that the predictions may be mediated by DNNs that share little overlap with biological vision. More problematically, we show that DNNs account for almost no results from psychological research. This contradicts the common claim that DNNs are good, let alone the best, models of human object recognition. We argue that theorists interested in developing biologically plausible models of human vision need to direct their attention to explaining psychological findings. More generally, theorists need to build models that explain the results of experiments that manipulate independent variables designed to test hypotheses rather than compete on making the best predictions. We conclude by briefly summarizing various promising modelling approaches that focus on psychological data.

show abstract

Section: The Empirical Problem With Claiming Dnns and Human Vision Ar...mentioning

confidence: 99%

Deep problems with neural network models of human vision

Bowers¹,

Dujmović²,

Montero³

et al. 2022

Behav Brain Sci

View full text Add to dashboard Cite

show abstract

“…In spite of these successes when tested on well known psychological phenomena, DNNs often fail on the most basic perceptual properties exhibited by humans. For example, DNNs do not possess human-like shape bias (Geirhos et al, 2018;Malhotra and Bowers, 2019), they appear to discriminate categories based on local instead of global features (Baker et al, 2018b;Malhotra et al, 2021), are much more susceptible to a low amount of image degradation (Geirhos et al, 2018), do not account for humans' similarity judgments of 3D shapes (German and Jacobs, 2020), and fail to support basic visual reasoning such as classifying images as the same or different (Puebla and Bowers, 2021). In addition, DNNs often act in surprising non-human-like ways, such as being fooled by adversarial images (Szegedy et al, 2013;Dujmović et al, 2020) and make bizarre classification errors to familiar objects in unusual poses (Kauderer-Abrams, 2017;Gong et al, 2014;Chen et al, 2017).…”

Section: Neural Network As a Model Of The Human Visual Systemmentioning

confidence: 99%

Mixed Evidence for Gestalt Grouping in Deep Neural Networks

Biscione¹,

Bowers²

2022

Preprint

View full text Add to dashboard Cite

Under some circumstances, humans tend to perceive individual elements as a group or 'whole'. This has been widely investigated for more than a century by the school of Gestalt Psychology, which formulated several laws of perceptual grouping. Recently, Deep Neural Networks (DNNs) trained on natural images have been proposed as compelling models of human vision based on reports that they learn internal representations similar to the primate ventral visual stream and show similar patterns of errors in object classification tasks. That is, DNNs often perform well on brain and behavioral benchmarks. Here we compared human and DNN responses in discrimination judgments that assess a range of Gestalt organization principles (Pomerantz et al., 1977;Pomerantz and Portillo, 2011). Amongst the DNNs tested we selected models that perform well on the Brain-Score benchmark (Schrimpf et al., 2018). We found that network trained on natural images exhibited sensitivity to shapes at the last stage of classification, which in some cases matched humans responses. When shape familiarity was controlled for (by using dot patterns that would not resemble shapes) we found the networks were insensitive to the standard Gestalt principles of proximity, orientation, and linearity, which have been shown to have a strong and robust effect on humans. This shows that models that perform well on behavioral and brain benchmarks nevertheless miss fundamental principles of human vision.

show abstract

“…On one hand, CNNs' internal activations are predictive of brain recordings in response to images taken from classic benchmark datasets in the mid and high levels of the ventral visual pathway (Yamins et al, 2013(Yamins et al, , 2014Cichy et al, 2016). On the other hand, CNNs show some behavioural discrepancies that undermine their plausibility as a model of the visual system: they are susceptible to adversarial attacks (Szegedy et al, 2013;Dujmović et al, 2020); they are highly susceptible to low amount of image degradation (Geirhos et al, 2018); they often classify images based on textures instead of shape (Geirhos et al, 2018) and local instead of global features (Baker et al, 2018), and do not account for humans' similarity judgments of 3D shapes (German and Jacobs, 2020). Within the framework of this debate, we aim to answer the question of whether CNNs can learn to support online invariance for a wide variety of transformations commonly experienced in the human visual environment.…”

Section: Introductionmentioning

confidence: 99%

Learning Online Visual Invariances for Novel Objects via Supervised and Self-Supervised Training

Biscione,

Bowers

2021

Preprint

View full text Add to dashboard Cite

CNNs can be trained to acquire online-invariance: the same internal representation is elicited following different transformations of the same, novel objects• We demonstrate on-line invariance for the following transformations: translation, rotation, scale, brightness, contrast, and viewpoint• Many different supervised networks acquire this property, as does a self-supervised network that solves the same/different task• As few as 50 images taken 10 object classes are needed to train for online invariance

show abstract

Can machine learning account for human visual object shape similarity judgments?

Cited by 13 publications

References 20 publications

Deep problems with neural network models of human vision

Deep problems with neural network models of human vision

Mixed Evidence for Gestalt Grouping in Deep Neural Networks

Learning Online Visual Invariances for Novel Objects via Supervised and Self-Supervised Training

Contact Info

Product

Resources

About