Deep Convolutional Neural Networks (CNNs) are gaining traction as the benchmark model of visual object recognition, with performance now surpassing humans. While CNNs can accurately assign one image to potentially thousands of categories, network performance could be the result of layers that are tuned to represent the visual shape of objects, rather than object category, since both are often confounded in natural images. Using two stimulus sets that explicitly dissociate shape from category, we correlate these two types of information with each layer of multiple CNNs. We also compare CNN output with fMRI activation along the human visual ventral stream by correlating artificial with neural representations. We find that CNNs encode category information independently from shape, peaking at the final fully connected layer in all tested CNN architectures. Comparing CNNs with fMRI brain data, early visual cortex (V1) and early layers of CNNs encode shape information. Anterior ventral temporal cortex encodes category information, which correlates best with the final layer of CNNs. The interaction between shape and category that is found along the human visual ventral pathway is echoed in multiple deep networks. Our results suggest CNNs represent category information independently from shape, much like the human visual system.

Some of the most impressive functional specialization in the human brain is found in occipitotemporal cortex (OTC), where several areas exhibit selectivity for a small number of visual categories, such as faces and bodies, and spatially cluster based on stimulus animacy.Previous studies suggest this animacy organization reflects the representation of an intuitive taxonomic hierarchy, distinct from the presence of face-and body-selective areas in OTC.Using human fMRI, we investigated the independent contribution of these two factors -the face-body division and taxonomic hierarchy -in accounting for the animacy organization of OTC, and whether they might also be reflected in the architecture of several deep neural networks. We found that graded selectivity based on animal resemblance to human faces and bodies masquerades as an apparent animacy continuum, which suggests that taxonomy is not a separate factor underlying the organization of the ventral visual pathway.

1Deep Convolutional Neural Networks (CNNs) are gaining traction as the benchmark model of 2 visual object recognition, with performance now surpassing humans. While CNNs can accurately 3 assign one image to potentially thousands of categories, network performance could be the result 4 of layers that are tuned to represent the visual shape of objects, rather than object category, since 5 both are often confounded in natural images. Using two stimulus sets that explicitly dissociate 6 shape from category, we correlate these two types of information with each layer of multiple 7CNNs. We also compare CNN output with fMRI activation along the human visual ventral 8 stream by correlating artificial with biological representations. We find that CNNs encode 9 category information independently from shape, peaking at the final fully connected layer in all 10 tested CNN architectures. Comparing CNNs with fMRI brain data, early visual cortex (V1) and 11 early layers of CNNs encode shape information. Anterior ventral temporal cortex encodes 12 category information, which correlates best with the final layer of CNNs. The interaction 13 between shape and category that is found along the human visual ventral pathway is echoed in 14 multiple deep networks. Our results suggest CNNs represent category information independently 15 from shape, much like the human visual system. 16 17

The ontogenetic development of human vision and the real-time neural processing of visual input exhibit a striking similarity—a sensitivity toward spatial frequencies that progresses in a coarse-to-fine manner. During early human development, sensitivity for higher spatial frequencies increases with age. In adulthood, when humans receive new visual input, low spatial frequencies are typically processed first before subsequent processing of higher spatial frequencies. We investigated to what extent this coarse-to-fine progression might impact visual representations in artificial vision and compared this to adult human representations. We simulated the coarse-to-fine progression of image processing in deep convolutional neural networks (CNNs) by gradually increasing spatial frequency information during training. We compared CNN performance after standard and coarse-to-fine training with a wide range of datasets from behavioral and neuroimaging experiments. In contrast to humans, CNNs that are trained using the standard protocol are very insensitive to low spatial frequency information, showing very poor performance in being able to classify such object images. By training CNNs using our coarse-to-fine method, we improved the classification accuracy of CNNs from 0% to 32% on low-pass-filtered images taken from the ImageNet dataset. The coarse-to-fine training also made the CNNs more sensitive to low spatial frequencies in hybrid images with conflicting information in different frequency bands. When comparing differently trained networks on images containing full spatial frequency information, we saw no representational differences. Overall, this integration of computational, neural, and behavioral findings shows the relevance of the exposure to and processing of inputs with variation in spatial frequency content for some aspects of high-level object representations.