A conceptual framework of computations in mid-level vision

Abstract

If a picture is worth a thousand words, as an English idiom goes, what should those words—or, rather, descriptors—capture? What format of image representation would be sufficiently rich if we were to reconstruct the essence of images from their descriptors? In this paper, we set out to develop a conceptual framework that would be: (i) biologically plausible in order to provide a better mechanistic understanding of our visual system; (ii) sufficiently robust to apply in practice on realistic images; and (iii) able to tap into underlying structure of our visual world. We bring forward three key ideas. First, we argue that surface-based representations are constructed based on feature inference from the input in the intermediate processing layers of the visual system. Such representations are computed in a largely pre-semantic (prior to categorization) and pre-attentive manner using multiple cues (orientation, color, polarity, variation in orientation, and so on), and explicitly retain configural relations between features. The constructed surfaces may be partially overlapping to compensate for occlusions and are ordered in depth (figure-ground organization). Second, we propose that such intermediate representations could be formed by a hierarchical computation of similarity between features in local image patches and pooling of highly-similar units, and reestimated via recurrent loops according to the task demands. Finally, we suggest to use datasets composed of realistically rendered artificial objects and surfaces in order to better understand a model's behavior and its limitations.

Which visual functions depend on intermediate visual regions? Insights from a case of developmental visual form agnosia

“…Could magnocellular transient processes suffice to support this ability, perhaps explaining how even in LG's case of hindered downstream flow, such "vision at a glance" might function normally? Interestingly, a recent framework for midlevel vision computations (Kubilius et al, 2014) proposes that the gist of the scene can be achieved in a pre-attentive manner by pooling together information from multiple sources (color, orientation, natural statistics of the scene and so on). For such a framework to support "vision at a glance" (Hochstein and Ahissar, 2002;Kubilius et al, 2014), it would be reasonable to assume that such pre-attentive computations should occur fast and based on coarse spatial resolution, as is the case with the magnocellular pathway (Schmolesky et al, 1998;Lamme and Roelfsema, 2000) which appears to function normally in LG (Gilaie-Dotan et al, 2009, 2011.…”

“…Interestingly, a recent framework for midlevel vision computations (Kubilius et al, 2014) proposes that the gist of the scene can be achieved in a pre-attentive manner by pooling together information from multiple sources (color, orientation, natural statistics of the scene and so on). For such a framework to support "vision at a glance" (Hochstein and Ahissar, 2002;Kubilius et al, 2014), it would be reasonable to assume that such pre-attentive computations should occur fast and based on coarse spatial resolution, as is the case with the magnocellular pathway (Schmolesky et al, 1998;Lamme and Roelfsema, 2000) which appears to function normally in LG (Gilaie-Dotan et al, 2009, 2011. Another such processing route that might also be normal in LG (see place recognition in Table 2), might rely on peripheral form-based computations, which are suggested to respond transiently (Gilaie-Dotan et al, 2008) and rely on coarse spatial resolution (Levy et al, 2001(Levy et al, , 2004Dumoulin and Wandell, 2008).…”

“…Gestalt grouping principles) have been hypothesized to be supported by the function of intermediate visual regions. But whether these are performed locally in V1 or together with V2/V3 early in the visual process ("the first/forward sweep"), or rely on contextual, top-down influences (as attention or past experience) from higher-order regions (Lamme and Roelfsema, 2000;Hochstein and Ahissar, 2002;Kubilius et al, 2014), remains unknown (for an overview see Brooks et al (2012)). Furthermore, even if these midlevel processes are computed in intermediate visual regions, whether they rely on a single mechanism, are performed in parallel in a similar fashion (e.g.…”

Foreground-background segmentation revealed during natural image viewing

“…Consequently, while computer science typically employs solutions that rely only seldom on previous neuroscientific knowledge, and its goal is to maximize task accuracy (e.g., with deep learning), visual neuroscience somehow lacks of solid computational models and formal explanations, ending up with several arbitrary assumptions in modeling, especially for mid-level vision processing, such as scene segmentation or shape features extraction (for a definition see: Kubilius J et al 2014).…”

Common spatiotemporal processing of visual features shapes object representation

“…First, the low-level description of the stimuli was grounded on features extracted by the early visual cortex (i.e., image contrast and spatial frequencies). Second, since shape is critical to interact with the surrounding environment 17 , we relied on a well-assessed, physiologically-motivated description of shape, i.e., the medial axis 18 . Finally, objects were also distinctively represented according to their superordinate categories.…”