The Ventral Visual Pathway Represents Animal Appearance over Animacy, Unlike Human Behavior and Deep Neural Networks

Abstract

Recent studies showed agreement between how the human brain and neural networks represent objects, suggesting that we might start to understand the underlying computations. However, we know that the human brain is prone to biases at many perceptual and cognitive levels, often shaped by learning history and evolutionary constraints. Here, we explore one such perceptual phenomenon, perceiving animacy, and use the performance of neural networks as a benchmark. We performed an fMRI study that dissociated object appearance (what an object looks like) from object category (animate or inanimate) by constructing a stimulus set that includes animate objects (e.g., a cow), typical inanimate objects (e.g., a mug), and, crucially, inanimate objects that look like the animate objects (e.g., a cow mug). Behavioral judgments and deep neural networks categorized images mainly by animacy, setting all objects (lookalike and inanimate) apart from the animate ones. In contrast, activity patterns in ventral occipitotemporal cortex (VTC) were better explained by object appearance: animals and lookalikes were similarly represented and separated from the inanimate objects. Furthermore, the appearance of an object interfered with proper object identification, such as failing to signal that a cow mug is a mug. The preference in VTC to represent a lookalike as animate was even present when participants performed a task requiring them to report the lookalikes as inanimate. In conclusion, VTC representations, in contrast to neural networks, fail to represent objects when visual appearance is dissociated from animacy, probably due to a preferred processing of visual features typical of animate objects.

Visual featural topography in the human ventral visual pathway

“…We are treating this “domain” structure as an observation to be explained instead of an explanatory theory because it is descriptive, vaguely defined, and does not offer hypothesis about exactly what information is represented here. The visual-feature-driven-by-action-mapping account not only explains this observation, but also makes predictions that are consistent with a series of results comparing the feature vs. domain effects in the literature: Objects that do not have prototypical shapes of a domain (e.g., a cup shapes like a cow) are processed by VOTC more similarly to items sharing its surface shape (e.g., animal in this case) and not to those in the same domain (regular cups) [41]; the animate-preferring areas are modulated by how “typical” (human-like) animals are [42]; features without domain contexts may still be able to produce effects [14,17,38]. Our supplementary analyses and feature-validation fMRI experiment analyses provided further support to this last point ( S1 Text; S7 and S8 Figs ): The featural effects were largely present when regressing out domain structure; The effect of right angle in bilateral medFG (aligning with the PPA) was present when the features are shown in isolation without object contexts and/or other features, and even during presentation of objects from non-preferring domains (i.e., when objects are small manipulable artifacts or animals).…”

Deep Neural Networks Point to Mid-level Complexity of Rodent Object Vision

“…While our comparisons with DNNs show that rodent object vision can be explained by convolutional layers, in primates fully connected layers of the same or similar network architectures capture perceived shape similarity better than convolutional layers (Kubilius et al, 2016; Kalfas et al, 2018; Bracci et al, 2019). This suggests that the rodent visual system is capable of a mid-level complexity of object processing which is markedly less complex than primates.…”

“…AlexNet (Krizhevsky et al, 2012) and VGG16 (Simonyan and Zisserman, 2014) were both taken from the MAT-LAB 2017b Deep Learning Toolbox and had been pre-trained on the ImageNet dataset (Russakovsky et al, 2015) to classify images into 1000 object categories. Both architectures have been extensively used to model ventral stream processing and object perception (Güçlü and van Gerven, 2015; Cadieu et al, 2014; Kalfas et al, 2018; Bracci et al, 2019; Kubilius et al, 2016). For the experiments involving videos we also used VGG11-C3D (called C3D in Tran et al, 2014), an architecture which is similar to VGG11 (Simonyan and Zisserman, 2014), but performs convolution and pooling across the two spatial and temporal dimensions (operating on 16-frame bins).…”

Rapid and dynamic processing of face pareidolia in the human brain

“…of an object vs. its semantic meaning [29][30][31][32][33][34] . Recently it was demonstrated that in ventral occipitotemporal cortex, objects that look like another object (for example, a cow-shaped mug) have BOLD activation patterns that are more similar to what object they look like (e.g., a cow) than to their actual object identity (e.g., a mug) 30 .…”

“…of an object vs. its semantic meaning [29][30][31][32][33][34] . Recently it was demonstrated that in ventral occipitotemporal cortex, objects that look like another object (for example, a cow-shaped mug) have BOLD activation patterns that are more similar to what object they look like (e.g., a cow) than to their actual object identity (e.g., a mug) 30 . Similarly, it has been debated whether object representations in ventral temporal cortex are organized by an overall semantic principle such as animacy 14 or real-world size 35,36 , or by visual properties that co-vary with these categories 29,31,32,34 .…”