Orthogonal Representations of Object Shape and Category in Deep Convolutional Neural Networks and Human Visual Cortex

Abstract

Deep Convolutional Neural Networks (CNNs) are gaining traction as the benchmark model of visual object recognition, with performance now surpassing humans. While CNNs can accurately assign one image to potentially thousands of categories, network performance could be the result of layers that are tuned to represent the visual shape of objects, rather than object category, since both are often confounded in natural images. Using two stimulus sets that explicitly dissociate shape from category, we correlate these two types of information with each layer of multiple CNNs. We also compare CNN output with fMRI activation along the human visual ventral stream by correlating artificial with neural representations. We find that CNNs encode category information independently from shape, peaking at the final fully connected layer in all tested CNN architectures. Comparing CNNs with fMRI brain data, early visual cortex (V1) and early layers of CNNs encode shape information. Anterior ventral temporal cortex encodes category information, which correlates best with the final layer of CNNs. The interaction between shape and category that is found along the human visual ventral pathway is echoed in multiple deep networks. Our results suggest CNNs represent category information independently from shape, much like the human visual system.

Training for object recognition with increasing spatial frequency: A comparison of deep learning with human vision

“…For the stimulus set of Bracci & Op de Beeck (2016), all three training regimes of MobileNet preferred shape over category. In addition, they also showed a smaller, yet significant correlation with the category-related behavioural similarities, as it was also shown before for other networks trained with unfiltered images by Zeman et al (2020). There are no meaningful effects of the training regime, so the extent and timing of training with low-pass filtered images does not affect the presence of a multi-feature representation in a CNN.…”

“…Bracci et al (2019) found that CNNs classify objects by animacy rather than appearance, and this bias towards animacy is also demonstrated in human judgements and brain representations. Zeman et al (2020) found that CNNs represent category independently from shape in object images, similarly to human object recognition areas. All together, these studies and more, have demonstrated the strength and breadth of applications in using CNNs as a model of human vision.…”

“…To detect any possible changes in representations due to our training protocol, we included two stimulus sets along with the associated CNN, behavioural, and neural similarity data from Bracci et al (2019) and Bracci et al (2016; with further analyses by Zeman et al, 2020) for comparison with our trained networks. In both studies, the authors dissociated appearance from the category for each of the stimuli, allowing for a controlled comparison between visual and more conceptual information (see Figures 2 and 3).…”

“…Stimuli (panel A) and behavioural models (panel B) from Zeman et al (2020). The rows in panel A represent stimuli that share category but differ in shape, whereas the columns represent stimuli that are similar in shape but belong to different categories.…”

Will We Ever Have Conscious Machines?

“…Furthermore, there is increasing evidence that, even though most artificial neural networks largely lack biological plausibility, they are nevertheless well-suited for modeling brain function. A number of recent studies have shown striking similarities in the processing and representational dynamics between artificial neural networks and the brain (Cichy et al, 2016;Zeman et al, 2020). For instance, in deep neural networks trained on visual object recognition, the spontaneous emergence of number detectors (Nasr et al, 2019), solid shape coding (Srinath et al, 2020), or center-periphery spatial organization (Mohsenzadeh et al, 2020) was observed.…”