Deep convolutional networks do not classify based on global object shape

Baker, Nicholas; Lu, Hongjing; Erlikhman, Gennady; Kellman, Philip J.

doi:10.1371/journal.pcbi.1006613

Cited by 322 publications

(291 citation statements)

References 38 publications

Supporting

Mentioning

245

Contrasting

Order By: Relevance

“…In another study 19 , the authors tested view-identity tuning on several CNN models to compare with their novel generative model (see below). Although they showed RSM results similar to ours (Figure 2), they incorporated a more sophisticated quantitative comparison with experimental data 15 23 . This implies that the goal of the face-processing system may not be merely classification.…”

Section: Discussionmentioning

confidence: 58%

“…For the macaque face-processing network, our study here could not find a CNN layer corresponding to the intermediate stage (ML). In addition, some recent studies have pointed out potential representational discrepancies between CNN and the ventral stream from behavioral consideration 19,23,[27][28][29] . (See also the related discussion on the 'computational gap' below.)…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

CNN explains tuning properties of anterior, but not middle, face-processing areas in macaque IT

Raman¹,

Hosoya²

2019

Preprint

View full text Add to dashboard Cite

Recent computational studies have emphasized layer-wise quantitative similarity between convolutional neural networks (CNNs) and the primate visual ventral stream. However, whether such similarity holds for the face-selective areas, a subsystem of the higher visual cortex, is not clear. Here, we extensively investigate whether CNNs exhibit tuning properties as previously observed in different macaque face areas. While simulating four past experiments on a variety of CNN models, we sought for the model layer that quantitatively matches the multiple tuning properties of each face area. Our results show that higher model layers explain reasonably well the properties of anterior areas, while no layer simultaneously explains the properties of middle areas, consistently across the model variation. Thus, the CNNs may have some similarity with the primate face-processing system in the near-goal representation, but differ in the intermediate computational process, thus requiring a more comprehensive model for understanding the entire system.Recently, the neuroscience community has witnessed the rise of the deep convolution neural network (CNN) 1 , a family of feedforward artificial neural networks, in computational modeling of the primate visual system. CNN models trained for behavioral goals have exhibited remarkable similarity to ventral visual areas in terms of stimulus-response relationship despite that the network itself was not directly optimized to fit neural data. For

show abstract

Section: Discussionmentioning

confidence: 58%

Section: Discussionmentioning

confidence: 99%

CNN explains tuning properties of anterior, but not middle, face-processing areas in macaque IT

Raman¹,

Hosoya²

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…In recent studies, Baker et al (2018) and Geirhos et al, (2018Geirhos et al, ( , 2019 reported that CNNs rely on local texture and shape features rather than global shape contours. For example, while keeping the contour intact, replacing the skin texture of a cat with that of an elephant resulted in Resnet-50 classifying the animal as an elephant (Geirhos et al, 2018).…”

Section: Discussionmentioning

confidence: 99%

“…Although CNNs are believed to explicitly represent object shapes in the higher layers (Kriegeskorte, 2015;LeCun et al, 2015;Kubilius et al, 2016), emerging evidence suggests that CNNs may largely use local texture patches to achieve successful object classification (Ballester & de Araujo, 2016, Gatys et al, 2017 or local rather than global shape contours for object recognition (Baker et al, 2018). In a recent demonstration, CNNs were found to be poor at classifying objects defined by silhouettes and edges, and when texture and shape cues were in conflict, classifying objects according to texture rather than shape cues (Geirhos et al, 2019; see also Baker et al, 2018). However, when Resnet-50 was trained with stylized ImageNet images in which the original texture of every single image was replaced with the style of a randomly chosen painting, object classification performance significantly improved, relied more on shape than texture cues, and became more robust to noise and image distortions (Geirhos et al, 2019).…”

Section: The Effect Of Training a Cnn On Original Vs Stylized Image-nmentioning

confidence: 99%

Limited correspondence in visual representation between the human brain and convolutional neural networks

Vaziri-Pashkam²

2020

Preprint

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) have achieved very high object categorization performance recently. It has increasingly become a common practice in human fMRI research to regard CNNs as working model of the human visual system. Here we reevaluate this approach by comparing fMRI responses from the human brain in three experiments with those from 14 different CNNs.Our visual stimuli included original and filtered versions of real-world object images and images of artificial objects. Replicating previous findings, we found a brain-CNN correspondence in a number of CNNs with lower and higher levels of visual representations in the human brain better resembling those of lower and higher CNN layers, respectively. Moreover, the lower layers of some CNNs could fully capture the representational structure of human early visual areas for both the original and filtered real-world object images. Despite these successes, no CNN examined could fully capture the representational structure of higher human visual processing areas. They also failed to capture that of artificial object images in all levels of visual processing. The latter is particularly troublesome, as decades of vision research has demonstrated that the same algorithms used in the processing of natural images would support the processing of artificial visual stimuli in the primate brain. Similar results were obtained when a CNN was trained with stylized object images that emphasized shape representation. CNNs likely represent visual information in fundamentally different ways from the human brain. Current CNNs thus may not serve as sound working models of the human visual system. Significance StatementRecent CNNs have achieved very high object categorization performance, with some even exceeding human performance. It has become common practice in recent neuroscience research to regard CNNs as working models of the human visual system. Here we evaluate this approach by comparing fMRI responses from the human brain with those from 14 different CNNs. Despite CNNs' ability to successfully perform visual object categorization like the human visual system, they appear to represent visual information in fundamentally different ways from the human brain. Current CNNs thus may not serve as sound working models of the human visual system. Given the current dominating trend of incorporating CNN modeling in visual neuroscience research, our results question the validity of such an approach.

show abstract

“…It might be that the CNN model in our study does not learn the same features that humans use to solve the task (also see Appendix "Kernels for first layer of CNN"). Baker et al (2018) find that convolutional neural networks trained on a large database of natural images (Russakovsky et al, 2015) use different image features than humans for object recognition. More specifically, Geirhos et al (2019) report that such convolutional neural network models mostly rely on texture to perform classifications, while humans rely more on object shape.…”

Section: Discussionmentioning

confidence: 99%

Constrained sampling from deep generative image models reveals mechanisms of human target detection

Fruend

2019

Preprint

View full text Add to dashboard Cite

The first steps of visual processing are often described as a bank of oriented filters followed by divisive normalization.This approach has been tremendously successful at predicting contrast thresholds in simple visual displays. However, it is unclear to what extent this kind of architecture also supports processing in more complex visual tasks performed in naturally looking images.We used a deep generative image model to embed arc segments with different curvatures in naturalistic images. These images contain the target as part of the image scene, resulting in considerable appearance variation of target as well as background. Three observers localized arc targets in these images, achieving an accuracy of 74.7% correct responses on average. Data were fit by several biologically inspired models, 4 standard deep convolutional neural networks (CNN) from the computer vision literature, and by a 5-layer CNN specifically trained for this task. Four models were particularly good at predicting observer responses, (i) a bank of oriented filters, similar to complex cells in primate area V1, (ii) a bank of oriented filters followed by tuned gain control, incorporating knowledge about cortical surround interactions, (iii) a bank of oriented filters followed by local normalization, (iv) the 5-layer specifically trained CNN. A control experiment with optimized stimuli based on these four models showed that the observers' data were best explained by model (ii) with tuned gain control.These data suggest that standard models of early vision provide good descriptions of performance in much more complex tasks than what they were designed for, while general purpose non-linear models such as convolutional neural networks do not.

show abstract

Deep convolutional networks do not classify based on global object shape

Cited by 322 publications

References 38 publications

CNN explains tuning properties of anterior, but not middle, face-processing areas in macaque IT

CNN explains tuning properties of anterior, but not middle, face-processing areas in macaque IT

Limited correspondence in visual representation between the human brain and convolutional neural networks

Constrained sampling from deep generative image models reveals mechanisms of human target detection

Contact Info

Product

Resources

About