The Visual System's Internal Model of the World

Abstract

The Bayesian paradigm has provided a useful conceptual theory for understanding perceptual computation in the brain. While the detailed neural mechanisms of Bayesian inference are not fully understood, recent computational and neurophysiological works have illuminated the underlying computational principles and representational architecture. The fundamental insights are that the visual system is organized as a modular hierarchy to encode an internal model of the world, and that perception is realized by statistical inference based on such internal model. In this paper, I will discuss and analyze the varieties of representational schemes of these internal models and how they might be used to perform learning and inference. I will argue for a unified theoretical framework for relating the internal models to the observed neural phenomena and mechanisms in the visual cortex.

Relating functional connectivity in V1 neural circuits and 3D natural scenes using Boltzmann machines

“…It provides some support to the tantalizing hypothesis that the visual cortex might be functioning like a generative model, e.g. a Boltzmann machine or Markov random field, for statistical inference (Lee and Mumford, 2003; Lee and Yuille, 2006; Lee, 2015)…”

Modality-independent effect of gravity in shaping the internal representation of 3D space

“…Perceptual distortions can be different in each modality and the consequences of removing gravity might also differ in a modality-specific manner. Hypothesis 2: gravity exerts its influence only after the available sensory inputs have already been integrated into a common, modality-independent representation of 3D space (Wolbers et al 2011, Loomis et al 2013, through which the brain interprets sensory signals to reconstruct a 3D representation of the perceived object (Curry 1972Kersten and Yuille 2003, Kersten et al 2004, Lee 2015. The sensory inputs associated to ρ and α that feed into this modality-independent representation might be distorted in a modality-specific fashion, but the act of removing gravity would affect only this internal representation, according to this hypothesis.…”

“…When only one sensory modality is available, as in our experiments, the weight given to the visual and haptic comparisons in the overall output would, however, reflect the fact that a channel that has been reconstructed is less reliable than one that relies on actual sensory inputs. Under this model and the premise that gravity facilitates the reconstruction of sensory information across modalities (Burns and Blohm 2010, Tagliabue and McIntyre 2011-2015, the consequences of removing gravity will be to decrease the weight given to information that is not actually sensed (i.e. is reconstructed) thus shifting the response patterns toward the sensory modality that is.…”

“…The weight of each separate comparison on the final response would depend on the expected reliability of signals within each representation, which in turn depends on the brain's ability to transform and reconstruct signals between representations. Based on studies of goaldirected movements (Tagliabue and McIntyre 2011, 2015, Arnoux et al 2017, gravity plays an important role in determining how reliably signals are transformed between sensory modalities, leading to predictable effects of removing gravity on shape perception. Out of the three possibilities presented here, logic and the quantitative assessments presented here support Hypothesis 3 as the most parsimonious explanation of the observed data.…”

“…Given its importance for interacting with the physical world, 3D object perception has been deeply investigated. Visual perception has received the most attention, showing how various features of the stimuli as disparities, size, occlusions, perspective, motion, shadows, shading, texture and blur all influence 3D visual perception (Welchman 2016) and how internal models constraint/shape the interpretation of the sensory signals (Curry 1972, Kersten and Yuille 2003, Kersten et al 2004, Lee 2015. Despite its critical importance to perception and action, visual perception suffers from measurable distortions: height underestimation with respect to width, also known as the horizontal-vertical, or "L", illusion (Avery andDay 1969, Hamburger andHansen 2010) and a systematic depth underestimation Philbeck 1999, Todd andNorman 2003).…”