Shape Selectivity of Middle Superior Temporal Sulcus Body Patch Neurons

Abstract

Functional MRI studies in primates have demonstrated cortical regions that are strongly activated by visual images of bodies. The presence of such body patches in macaques allows characterization of the stimulus selectivity of their single neurons. Middle superior temporal sulcus body (MSB) patch neurons showed similar stimulus selectivity for natural, shaded, and textured images compared with their silhouettes, suggesting that shape is an important determinant of MSB responses. Here, we examined and modeled the shape selectivity of single MSB neurons. We measured the responses of single MSB neurons to a variety of shapes producing a wide range of responses. We used an adaptive stimulus sampling procedure, selecting and modifying shapes based on the responses of the neuron. Forty percent of shapes that produced the maximal response were rated by humans as animal-like, but the top shape of many MSB neurons was not judged as resembling a body. We fitted the shape selectivity of MSB neurons with a model that parameterizes shapes in terms of curvature and orientation of contour segments, with a pixel-based model, and with layers of units of convolutional neural networks (CNNs). The deep convolutional layers of CNNs provided the best goodness-of-fit, with a median explained explainable variance of the neurons’ responses of 77%. The goodness-of-fit increased along the convolutional layers’ hierarchy but was lower for the fully connected layers. Together with demonstrating the successful modeling of single unit shape selectivity with deep CNNs, the data suggest that semantic or category knowledge determines only slightly the single MSB neuron’s shape selectivity.

Deep Neural Networks Point to Mid-level Complexity of Rodent Object Vision

“…Recently, a family of computational models has emerged in the form of convolutional deep neural networks (DNNs) that allow to simulate this hierarchical information processing. When trained on object recognition, DNNs show interesting commonalities with the primate ventral stream, with a progression of representations that is surprisingly similar to what is seen in monkeys and humans for brief stimulus presentations (Cadieu et al, 2014; Yamins et al, 2014; Güçlü and van Gerven, 2015; Kalfas et al, 2017, 2018; Pospisil et al, 2018; Bashivan et al, 2019), as such capturing important aspects of object recognition and perceived shape similarity (Yamins et al, 2014; Kubilius et al, 2016; Kalfas et al, 2018). The architecture of these computational models is composed of series of convolutional layers that perform local filtering operations, followed by fully connected layers, which gradually transforms pixel level inputs into a high-level representational space where object categories are linearly separable.…”

The ventral visual pathway represents animal appearance over animacy, unlike human behavior and deep neural networks

“…The animacy division is considered one of the main organizational principles in visual cortex (e.g., Grill-Spector and Weiner, 2014), but information content underlying this division is highly debated (Baldassi et al, 2013; Grill-Spector and Weiner, 2014; Nasr et al, 2014; Bracci and Op de Beeck, 2016; Bracci et al, 2017b; Kalfas et al, 2017). Animacy and other category distinctions are often correlated with a range of low- and higher-level visual features such as the spatial frequency spectrum (Nasr et al, 2014; Rice et al, 2014) and shape (Cohen et al, 2014; Jozwik et al, 2016), but the animacy structure remains even when dissociated from such features (Bracci and Op de Beeck, 2016).…”

“…A fundamental goal in visual neuroscience is to reach a deep understanding of the neural code underlying object representations; how does the brain represent objects we perceive around us? Over the years, research has characterized object representations in the primate brain in terms of their content for a wide range of visual and semantic object properties such as shape, size, or animacy (Konkle and Oliva, 2012; Nasr et al, 2014; Bracci and Op de Beeck, 2016; Kalfas et al, 2017). More recently, our understanding of these multidimensional object representations has been lifted to a higher level by the advent of so-called deep-convolutional neural networks (DNNs) that do not only reach human behavioral performance in image categorization (Russakovsky et al, 2014; He et al, 2015; Kheradpisheh et al, 2016a), but also appear to develop representations that share many of the properties of primate object representations (Cadieu et al, 2014; Guclu and van Gerven, 2014; Khaligh-Razavi and Kriegeskorte, 2014; Yamins et al, 2014; Guclu and van Gerven, 2015; Kubilius et al, 2016).…”

The Ventral Visual Pathway Represents Animal Appearance over Animacy, Unlike Human Behavior and Deep Neural Networks

“…A fundamental goal in visual neuroscience is to reach a deep understanding of the neural code underlying object representa-tions-how does the brain represent objects we perceive around us? Over the years, research has characterized object representations in the primate brain in terms of their content for a wide range of visual and semantic object properties such as shape, size, or animacy (Konkle and Oliva, 2012;Nasr et al, 2014;Bracci and Op de Beeck, 2016;Kalfas et al, 2017). More recently, our understanding of these multidimensional object representations has been lifted to a higher level by the advent of so-called deep-convolutional neural networks (DNNs) that do not only reach human behavioral performance in image categorization (Russakovsky et al, 2014;He et al, 2015;Kheradpisheh et al, 2016a), but also appear to develop representations that share many of the properties of primate object representations (Cadieu et al, 2014;Güçlü and van Gerven, 2014;Khaligh-Razavi and Kriegeskorte, 2014;Yamins et al, 2014;Güçlü and van Gerven, 2015;Kubilius et al, 2016).…”