Looking for Image Statistics: Active Vision With Avatars in a Naturalistic Virtual Environment

Abstract

The efficient coding hypothesis posits that sensory systems are tuned to the regularities of their natural input. The statistics of natural image databases have been the topic of many studies, which have revealed biases in the distribution of orientations that are related to neural representations as well as behavior in psychophysical tasks. However, commonly used natural image databases contain images taken with a camera with a planar image sensor and limited field of view. Thus, these images do not incorporate the physical properties of the visual system and its active use reflecting body and eye movements. Here, we investigate quantitatively, whether the active use of the visual system influences image statistics across the visual field by simulating visual behaviors in an avatar in a naturalistic virtual environment. Images with a field of view of 120° were generated during exploration of a virtual forest environment both for a human and cat avatar. The physical properties of the visual system were taken into account by projecting the images onto idealized retinas according to models of the eyes' geometrical optics. Crucially, different active gaze behaviors were simulated to obtain image ensembles that allow investigating the consequences of active visual behaviors on the statistics of the input to the visual system. In the central visual field, the statistics of the virtual images matched photographic images regarding their power spectra and a bias in edge orientations toward cardinal directions. At larger eccentricities, the cardinal bias was superimposed with a gradually increasing radial bias. The strength of this effect depends on the active visual behavior and the physical properties of the eye. There were also significant differences between the upper and lower visual field, which became stronger depending on how the environment was actively sampled. Taken together, the results show that quantitatively relating natural image statistics to neural representations and psychophysical behavior requires not only to take the structure of the environment into account, but also the physical properties of the visual system, and its active use in behavior.

Holistic inference explains human perception of stimulus orientation

“…Previous studies have shown that these statistics are relatively robust with regard to the specific methods they were measured with and the image content of the natural scenes they were computed for, showing characteristic peaks at both cardinal orientations (Coppola et al, 1998; Girshick et al, 2011; Wang et al, 2016). However, outdoor scenes containing fewer man-made objects typically show less pronounced peaks at the cardinals compared to indoor scenes (Coppola et al, 1998; Straub and Rothkopf,2021) (see Supplementary Fig. 2).…”

“…However, outdoor scenes containing fewer man-made objects typically show less pronounced peaks at the cardinals compared to indoor scenes (Coppola et al, 1998;Straub and Rothkopf, 2021) (see Supplementary Fig. 2).…”

What and Where: Location-Dependent Feature Sensitivity as a Canonical Organizing Principle of the Visual System

“…Incorporating topographic variations in feature encodings may allow neural networks to access priors contained in natural “embodied” visual experience ( Mineault et al, 2021 ), built over million years of species-specific evolution. This would not only bring in silico models closer in line with in vivo visual systems, but it may also bridge the gap between species-specific visual inputs and artificial training sets, which lack the structure of natural embodied visual experience ( Straub and Rothkopf, 2021 ). Alternatively, the use of more naturalistic “animal-view” movies as input ( Betsch et al, 2004 ), coupled with architectural flexibility, may allow neural networks to learn topographic specializations predictive of patterns in the retina or other hierarchical layers of the visual system ( Doshi and Konkle, 2021 ; Blauch et al, 2022 ).…”

Editorial: Active Vision and Perception in Human-Robot Collaboration

“…Finding the underlying design principles which allow humans to adaptively find and select relevant information (Tistarelli and Sandini, 1993 ; Findlay and Gilchrist, 2003 ; Krause and Guestrin, 2007 ; Friston et al, 2015 ; Ognibene and Baldassare, 2015 ; Bajcsy et al, 2017 ; Jayaraman and Grauman, 2018 ; Ballard and Zhang, 2021 ) is important for Robotics and related fields (Shimoda et al, 2021 ; Straub and Rothkopf, 2021 ). Active inference, which has recently become influential in computational neuroscience, is a normative framework proposing one such principle: action, perception, and learning are the result of minimization of variational free energy, a form of prediction error.…”