Natural Scene Categories Revealed in Distributed Patterns of Activity in the Human Brain

Abstract

Human subjects are extremely efficient at categorizing natural scenes, despite the fact that different classes of natural scenes often share similar image statistics. Thus far, however, it is unknown where and how complex natural scene categories are encoded and discriminated in the brain. We used functional magnetic resonance imaging (fMRI) and distributed pattern analysis to ask what regions of the brain can differentiate natural scene categories (such as forests vs mountains vs beaches). Using completely different exemplars of six natural scene categories for training and testing ensured that the classification algorithm was learning patterns associated with the category in general and not specific exemplars. We found that area V1, the parahippocampal place area (PPA), retrosplenial cortex (RSC), and lateral occipital complex (LOC) all contain information that distinguishes among natural scene categories. More importantly, correlations with human behavioral experiments suggest that the information present in the PPA, RSC, and LOC is likely to contribute to natural scene categorization by humans. Specifically, error patterns of predictions based on fMRI signals in these areas were significantly correlated with the behavioral errors of the subjects. Furthermore, both behavioral categorization performance and predictions from PPA exhibited a significant decrease in accuracy when scenes were presented up-down inverted. Together these results suggest that a network of regions, including the PPA, RSC, and LOC, contribute to the human ability to categorize natural scenes.

Similarity representation of pattern-information fMRI

“…In 2011, Kravitz et al [23] applied MDS to reconstruct and visualize representational dissimilarity structure derived from the distributed neural responses to 96 diverse real-world scenes. The authors found, contrary to previous studies [29,30], that representations in the ventral temporal parahippocampal place area (PPA) were characterized primarily by the spatial factor of expanse (open, closed) and in early visual cortex (EVC) primarily by distance (near, far), not by category or context. Kriegeskorte et al [22] applied hierarchical clustering analysis of response patterns in human brain evoked by 92 ungrouped-object stimuli photos, and found that object representation was inherently categorical in inferotemporal cortex (IT): animate and inanimate objects form the two major clusters; faces and bodies form subclusters within the animate cluster.…”

A neural basis for real-world visual search in human occipitotemporal cortex

“…Forty-eight different perceptual masks were created. Each was a colored picture of a mixture of white noise at different spatial frequencies on which a naturalistic texture was superimposed (31). Within a scanning session each of the presented pictures was unique.…”

Simple line drawings suffice for functional MRI decoding of natural scene categories

“…We have previously found that information about scene category is contained in patterns of fMRI activity in the parahippocampal place area (PPA), the retrosplenial cortex (RSC), the lateral occipital complex (LOC), and the primary visual cortex (V1) (12). PPA activity patterns appear to be linked most closely to human behavior (12), and activity in V1, PPA, and RSC elicited by good exemplars of scene categories contains significantly more scene category-specific information than patterns elicited by bad exemplars do (13).…”

“…PPA activity patterns appear to be linked most closely to human behavior (12), and activity in V1, PPA, and RSC elicited by good exemplars of scene categories contains significantly more scene category-specific information than patterns elicited by bad exemplars do (13). Interestingly, inspection of the average images of good and bad exemplars of a category suggests that good exemplars may contain more defined global structure apparent in the images than bad exemplars (13), suggesting that features that capture global information in the image may play a particularly important role in scene categorization.…”

“…Visual areas V2, VP (known as V3v, for ventral V3, in some nomenclatures), and V4 are of interest because they build more complex representations based on the information in V1, including representations that rely on more global aspects of the image (16)(17)(18). The PPA and the RSC, which have been shown to prefer scenes over objects and other visual stimuli (19,20), are of interest in this analysis because they (and, to some extent, the LOC) contain information about scene category in their activity patterns that are closely linked to behavioral scene-categorization performance (12).…”