Generative Models for Active Vision

Abstract

The active visual system comprises the visual cortices, cerebral attention networks, and oculomotor system. While fascinating in its own right, it is also an important model for sensorimotor networks in general. A prominent approach to studying this system is active inference—which assumes the brain makes use of an internal (generative) model to predict proprioceptive and visual input. This approach treats action as ensuring sensations conform to predictions (i.e., by moving the eyes) and posits that visual percepts are the consequence of updating predictions to conform to sensations. Under active inference, the challenge is to identify the form of the generative model that makes these predictions—and thus directs behavior. In this paper, we provide an overview of the generative models that the brain must employ to engage in active vision. This means specifying the processes that explain retinal cell activity and proprioceptive information from oculomotor muscle fibers. In addition to the mechanics of the eyes and retina, these processes include our choices about where to move our eyes. These decisions rest upon beliefs about salient locations, or the potential for information gain and belief-updating. A key theme of this paper is the relationship between “looking” and “seeing” under the brain's implicit generative model of the visual world.

Active Inference in Robotics and Artificial Agents: Survey and Challenges

“…Moreover, in a discrete state space setting, it is often tractable to explicitly evaluate the integral over time and compute the optimal plan posterior since agents in discrete states typically exist in small enough environments such that all policies can be explicitly enumerated and evaluated [71]. Discrete state space active inference has been widely applied in computational neuroscience to simulate choice behaviour, e.g., saccades [76] and exploratory behaviour [68].…”

A Bayesian computational model to investigate expert anticipation of a seemingly unpredictable ball bounce

“…Active inference extends predictive coding models of perception to the use of action to minimize future prediction errors (Friston, 2005;Parr & Friston, 2018. In addition to revising their generative model, a Bayesian agent can minimize prediction errors through movements (e.g., see the use of vision to minimize surprisal; Parr et al, 2021) or can actively change the world into the predicted state (Adams et al, 2013;Sarpeshkar et al, 2017). Harris et al (2021) have recently suggested that active inference can enhance our understanding of skilled anticipation by providing a principled account of how actions are used to optimize predictions, as well as accounting for decision making via Bayesian inference.…”

“…An additional benefit of Bayesian brain and active inference frameworks is that they are rooted in computational models of perceptual processes (Adams et al, 2015;Parr et al, 2021;Smith et al, 2020). These models formalise active inference, and the process of Bayesian belief updating, making mechanistic explanations potentially clear and testable.…”

“…However, it remains unclear exactly how experts are able to optimally integrate and weight multiple, everchanging, and often incomplete, information sources to make effective anticipatory judgements (Cañal-Bruland & Mann, 2015;Gredin et al, 2018;Williams & Jackson, 2019). In this study, we explore how performers weight the importance of different information sources when anticipating the bounce of a ball using a Bayesian computational model based on active inference theories of perception and action (Parr et al, 2021;Parr & Friston, 2019).…”

Disentangling What and Where for 3D Object-Centric Representations Through Active Inference

“…the variational free energy [5]. In [18], Parr et al propose a model for (human) vision, which considers a scene as a factorization of separate (parts of) objects, encoding their identity, scale and pose. This is in line with the so called two stream hypothesis, which states that visual information is processed by a dorsal ("where") stream on the one hand, representing where an object is in the space, and a ventral ("what") stream on the other hand, representing object identity [4].…”