Object segmentation controls image reconstruction from natural scenes

Abstract

The structure of the physical world projects images onto our eyes. However, those images are often poorly representative of environmental structure: well-defined boundaries within the eye may correspond to irrelevant features of the physical world, while critical features of the physical world may be nearly invisible at the retinal projection. The challenge for the visual cortex is to sort these two types of features according to their utility in ultimately reconstructing percepts and interpreting the constituents of the scene. We describe a novel paradigm that enabled us to selectively evaluate the relative role played by these two feature classes in signal reconstruction from corrupted images. Our measurements demonstrate that this process is quickly dominated by the inferred structure of the environment, and only minimally controlled by variations of raw image content. The inferential mechanism is spatially global and its impact on early visual cortex is fast. Furthermore, it retunes local visual processing for more efficient feature extraction without altering the intrinsic transduction noise. The basic properties of this process can be partially captured by a combination of small-scale circuit models and large-scale network architectures. Taken together, our results challenge compartmentalized notions of bottom-up/top-down perception and suggest instead that these two modes are best viewed as an integrated perceptual mechanism.

Contrast versus identity encoding in the face image follow distinct orientation selectivity profiles

“…Two recent works showed that the orientationselective processing of Gabor patches sharpens under the influence of the higher-level representation of the natural scene in which it is inserted [33,34]. While Teufel and colleagues [33] suggest that such modulations stem from the high-level semantic interpretation of the image, Neri [34] interprets his findings as arising from mid-level stimulus analyses, as they were not modulated by manipulations hampering stimulus semantic interpretation. However, we need to be cautious while comparing our findings to those previous works.…”

“…For example, access to a higher-level representation has been shown to boost or hamper the processing of basic attributes, depending on whether a stimulus is near or supra-threshold [60][61][62][63]. Two recent works showed that the orientationselective processing of Gabor patches sharpens under the influence of the higher-level representation of the natural scene in which it is inserted [33,34]. While Teufel and colleagues [33] suggest that such modulations stem from the high-level semantic interpretation of the image, Neri [34] interprets his findings as arising from mid-level stimulus analyses, as they were not modulated by manipulations hampering stimulus semantic interpretation.…”

“…Another important aspect is that in both the Neri [34] and Teufel et al [33] studies, participants were instructed to detect and extract the orientation of a local narrowband Gabor patch inserted either on the contour of a shape, or on an irrelevant contour. This stimulus-task combination involved processing of shape boundaries, likely encouraging higher-level influences.…”

“…In the case of a functional link between primary and high-level orientation dependencies, we further hypothesized to observe a difference in the orientation selectivity profiles in the contrast detection task for upright and inverted faces. Considering past findings that high-level representations influence the primary processing of oriented contrast [33,34], higher-level feedback about the presence of an upright face in the image plausibly modulates the orientation tuning towards the diagnostic orientation already at the primary stages of processing and reduces the horizontal effect compared to when an inverted face is processed.…”

“…There is also evidence for higher-level influences on the primary encoding of local orientation. For example, the sensitivity to the orientation of a local Gabor embedded in a complex scene is influenced by the figure-ground organization of the scene at mid-or high-level processing stages more than by the low-level contrast properties at the locus of Gabor insertion [33,34].…”