Orthogonal Representations of Object Shape and Category in Deep Convolutional Neural Networks and Human Visual Cortex

Abstract

1Deep Convolutional Neural Networks (CNNs) are gaining traction as the benchmark model of 2 visual object recognition, with performance now surpassing humans. While CNNs can accurately 3 assign one image to potentially thousands of categories, network performance could be the result 4 of layers that are tuned to represent the visual shape of objects, rather than object category, since 5 both are often confounded in natural images. Using two stimulus sets that explicitly dissociate 6 shape from category, we correlate these two types of information with each layer of multiple 7CNNs. We also compare CNN output with fMRI activation along the human visual ventral 8 stream by correlating artificial with biological representations. We find that CNNs encode 9 category information independently from shape, peaking at the final fully connected layer in all 10 tested CNN architectures. Comparing CNNs with fMRI brain data, early visual cortex (V1) and 11 early layers of CNNs encode shape information. Anterior ventral temporal cortex encodes 12 category information, which correlates best with the final layer of CNNs. The interaction 13 between shape and category that is found along the human visual ventral pathway is echoed in 14 multiple deep networks. Our results suggest CNNs represent category information independently 15 from shape, much like the human visual system. 16 17

Untangling the animacy organization of occipitotemporal cortex

“…Most network architectures consist of numerous convolutional layers, which tend to abstract local feature information when trained on large stimulus sets, followed by a few fully connected (FC) layers that more closely mirror the category structure in the training set. Several studies have reported correlations between FC layers and category-selective areas in human OTC suggesting that the animacy division is represented in these networks when trained on natural images (Bracci et al 2019;Jozwik et al 2017; Khaligh-Razavi and Kriegeskorte, 2014;Zeman et al 2020). However, these studies did not include a taxonomic hierarchy in their stimulus designs.…”

“…Third, we investigated whether face and body morphology -in particular, the relative similarity of animal faces and bodies to those of humans -might better explain the relationship between activity patterns than taxonomy. Finally, we assessed whether both factors might be reflected in the patterns of activation weights of layers of multiple deep neural networks (DNNs), which some studies suggest also distinguish the animacy of objects (Bracci et al 2019;Jozwik et al 2017; Khaligh-Razavi and Kriegeskorte, 2014;Zeman et al 2020).…”

Diverse deep neural networks all predict human IT well, after training and fitting

“…Despite being developed as computer vision tools, DNNs trained to recognise objects in images are also unsurpassed at predicting how natural images are represented in high-level ventral visual areas of the human and non-human primate brain (Agrawal et al, 2014;Bashivan et al, 2019;Cadieu et al, 2014;Cichy et al, 2016;Devereux et al, 2018;Eickenberg et al, 2017;Güçlü & van Gerven, 2015;Horikawa & Kamitani, 2017;Kubilius et al, 2018;Lindsay, 2020;Ponce et al, 2019;Schrimpf et al, 2018;Xu & Vaziri-Pashkam, 2020;Yamins & DiCarlo, 2016). There is some variability in the accuracy with which different recent DNNs can predict high-level visual representations Xu & Vaziri-Pashkam, 2020;Zeman et al, 2020), despite broadly high performance. It remains unclear how strongly network design choices, such as depth, architecture, task training, and subsequent model fitting to neural data may contribute to the observed variations.…”

“…Agrawal et al, 2014;Bashivan et al, 2019;Cadieu et al, 2014;Güçlü & van Gerven, 2015;Horikawa & Kamitani, 2017;Ponce et al, 2019;, while others treat the representations within a layer of a network as fixed (e.g. Truzzi & Cusack, 2020;Zeman et al, 2020).…”

“…graded similarity between DNN representations and the human ventral stream(Güçlü & van Gerven, 2015;Xu & Vaziri-Pashkam, 2020;Yamins et al, 2014;Zeman et al, 2020), and suggests that the learning of natural image features of the correct complexity is responsible for the superior performance oflater layers, rather than inherent architectural properties such as the spatial scale of the representations. Most models show a sharp decline in hIT match towards the final output layers, likely because their training target, a sparse vector indicating which of 1,000 objects possible within the Imagenet ILSVRC challenge dataset is present in an image, forces late layers not to represent graded similarities between images.…”