Simple line drawings suffice for functional MRI decoding of natural scene categories

Abstract

Humans are remarkably efficient at categorizing natural scenes. In fact, scene categories can be decoded from functional MRI (fMRI) data throughout the ventral visual cortex, including the primary visual cortex, the parahippocampal place area (PPA), and the retrosplenial cortex (RSC). Here we ask whether, and where, we can still decode scene category if we reduce the scenes to mere lines. We collected fMRI data while participants viewed photographs and line drawings of beaches, city streets, forests, highways, mountains, and offices. Despite the marked difference in scene statistics, we were able to decode scene category from fMRI data for line drawings just as well as from activity for color photographs, in primary visual cortex through PPA and RSC. Even more remarkably, in PPA and RSC, error patterns for decoding from line drawings were very similar to those from color photographs. These data suggest that, in these regions, the information used to distinguish scene category is similar for line drawings and photographs. To determine the relative contributions of local and global structure to the human ability to categorize scenes, we selectively removed long or short contours from the line drawings. In a category-matching task, participants performed significantly worse when long contours were removed than when short contours were removed. We conclude that global scene structure, which is preserved in line drawings, plays an integral part in representing scene categories.

Coding of navigational affordances in the human visual system

“…We were surprised that we did not observe reliable evidence for navigational-affordance coding in the PPA, which has long been hypothesized to represent the spatial layout of scenes (8,(12)(13)(14). One possibility is that the PPA is sensitive to the coarse shape of the local environment as defined by the walls, but is insensitive to fine-grained manipulations of spatial structure as defined by the locations of exits.…”

Visual Representation of Contour and Shape

“…This makes good sense: the presence of a contour signals some physically significant "event" in the world-whether it be the occluding boundary of an object, a reflectance change, or something else. Indeed, human observers are just as good at scene recognition with line drawings as they are with full-color photographs (e.g., Walther et al, 2011). Similarly, object recognition (e.g., Biederman & Ju, 1988) and 3D shape perception (e.g., Cole et al, 2009) are often just as good with line drawings as they are with shaded images.…”

How do humans sketch objects?

“…Such pictographs predate the appearance of language by tens of thousands of years and today the ability to draw and recognize sketched objects is ubiquitous. In fact, recent * e-mail: m.eitz@tu-berlin.de † e-mail: hays@cs.brown.edu ‡ e-mail: marc.alexa@tu-berlin.de neuroscience work suggests that simple, abstracted sketches activate our brain in similar ways to real stimuli [Walther et al 2011]. Despite decades of graphics research, sketching is the only mechanism for most people to render visual content.…”