Model-Based Approaches to Active Perception and Control

Pezzulo, Giovanni; Rigoli, Francesco; Iodice, Pierpaolo; Maisto, Domenico; Stoianov, Ivilin

doi:10.3390/e19060266

Cited by 42 publications

(34 citation statements)

References 139 publications

(244 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Active inference also provides a suitable framework for investigating the emergence of action-oriented models. Previous work has highlighted the fact that active inference is consistent with, and necessarily prescribes, frugal and parsimonious generative models, thus providing a potential bridge between 'representation-hungry' approaches to cognition espoused by classical cognitivism and the 'representation-free' approaches advocated by embodied and enactive approaches [6,12,13,64,[67][68][69][70][71][72][73][74][75].…”

Section: Active Inferencementioning

confidence: 99%

See 1 more Smart Citation

Learning action-oriented models through active inference

2020

View full text Add to dashboard Cite

Converging theories suggest that organisms learn and exploit probabilistic models of their environment. However, it remains unclear how such models can be learned in practice. The open-ended complexity of natural environments means that it is generally infeasible for organisms to model their environment comprehensively. Alternatively, action-oriented models attempt to encode a parsimonious representation of adaptive agent-environment interactions. One approach to learning action-oriented models is to learn online in the presence of goal-directed behaviours. This constrains an agent to behaviourally relevant trajectories, reducing the diversity of the data a model need account for. Unfortunately, this approach can cause models to prematurely converge to sub-optimal solutions, through a process we refer to as a bad-bootstrap. Here, we exploit the normative framework of active inference to show that efficient action-oriented models can be learned by balancing goal-oriented and epistemic (information-seeking) behaviours in a principled manner. We illustrate our approach using a simple agent-based model of bacterial chemotaxis. We first demonstrate that learning via goal-directed behaviour indeed constrains models to behaviorally relevant aspects of the environment, but that this approach is prone to sub-optimal convergence. We then demonstrate that epistemic behaviours facilitate the construction of accurate and comprehensive models, but that these models are not tailored to any specific behavioural niche and are therefore less efficient in their use of data. Finally, we show that active inference agents learn models that are parsimonious, tailored to action, and which avoid bad bootstraps and sub-optimal convergence. Critically, our results indicate that models learned through active inference can support adaptive behaviour in spite of, and indeed because of, their departure from veridical representations of the environment. Our approach provides a principled method for learning adaptive models from limited interactions with an environment, highlighting a route to sample efficient learning algorithms.

show abstract

Section: Active Inferencementioning

confidence: 99%

“…One approach to this problem is for organisms to selectively model their world in a way that supports action [9][10][11][12][13][14]. We refer to such models as action-oriented, as their functional purpose is to enable adaptive behaviour, rather than to represent the world in a complete or accurate manner.…”

Section: Introductionmentioning

confidence: 99%

Learning action-oriented models through active inference

2020

View full text Add to dashboard Cite

show abstract

“…Recent findings suggest that the role of hippocampus in goal-directed navigation may be mediated by the strong projections from the hippocampal CA1 and subicular areas to the ventral striatum (vStr) [ 6 ], which might convey spatial-contextual information and permit forming place-reward associations [ 7 – 10 ]. From a computational perspective, the hippocampus and ventral striatum may jointly implement a model-based controller for goal-directed choice [ 8 , 11 – 17 ]. In this scheme, HC and vStr might be mapped to the two essential components of a model-based reinforcement learning (MB-RL) controller [ 1 , 18 ]: the state-transition model , which is essentially a model of the task that permits to predict the next location given the current state (say, a given place) and chosen action, and the state-value model , which encodes the (expected) reward associated to each state, respectively.…”

Section: Introductionmentioning

confidence: 99%

Model-based spatial navigation in the hippocampus-ventral striatum circuit: A computational analysis

et al. 2018

View full text Add to dashboard Cite

While the neurobiology of simple and habitual choices is relatively well known, our current understanding of goal-directed choices and planning in the brain is still limited. Theoretical work suggests that goal-directed computations can be productively associated to model-based (reinforcement learning) computations, yet a detailed mapping between computational processes and neuronal circuits remains to be fully established. Here we report a computational analysis that aligns Bayesian nonparametrics and model-based reinforcement learning (MB-RL) to the functioning of the hippocampus (HC) and the ventral striatum (vStr)–a neuronal circuit that increasingly recognized to be an appropriate model system to understand goal-directed (spatial) decisions and planning mechanisms in the brain. We test the MB-RL agent in a contextual conditioning task that depends on intact hippocampus and ventral striatal (shell) function and show that it solves the task while showing key behavioral and neuronal signatures of the HC—vStr circuit. Our simulations also explore the benefits of biological forms of look-ahead prediction (forward sweeps) during both learning and control. This article thus contributes to fill the gap between our current understanding of computational algorithms and biological realizations of (model-based) reinforcement learning.

show abstract

“…More precisely, since this quantity is not directly accessible by an agent, it is thought that the variational free energy (an upper bound to sensory surprisal) is minimised in its place. In active inference, perceptual and motor processes are often described as entangled and inseparable [25]- [27] providing thus a new possible methodology combining estimation and control following embodied/enactive theories of the mind. We previously presented a conceptual account of active inference and its role for nonmodular architectures of cognitive systems [14].…”

Section: Introductionmentioning

confidence: 99%

Nonmodular Architectures of Cognitive Systems based on Active Inference

Baltieri

Buckley

2019

2019 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

In psychology and neuroscience it is common to describe cognitive systems as input/output devices where perceptual and motor functions are implemented in a purely feedforward, open-loop fashion. On this view, perception and action are often seen as encapsulated modules with limited interaction between them. While embodied and enactive approaches to cognitive science have challenged the idealisation of the brain as an input/output device, we argue that even the more recent attempts to model systems using closed-loop architectures still heavily rely on a strong separation between motor and perceptual functions. Previously, we have suggested that the mainstream notion of modularity strongly resonates with the separation principle of control theory. In this work we present a minimal model of a sensorimotor loop implementing an architecture based on the separation principle. We link this to popular formulations of perception and action in the cognitive sciences, and show its limitations when, for instance, external forces are not modelled by an agent. These forces can be seen as variables that an agent cannot directly control, i.e., a perturbation from the environment or an interference caused by other agents. As an alternative approach inspired by embodied cognitive science, we then propose a nonmodular architecture based on the active inference framework. We demonstrate the robustness of this architecture to unknown external inputs and show that the mechanism with which this is achieved in linear models is equivalent to integral control.

show abstract

Model-Based Approaches to Active Perception and Control

Cited by 42 publications

References 139 publications

Learning action-oriented models through active inference

Learning action-oriented models through active inference

Model-based spatial navigation in the hippocampus-ventral striatum circuit: A computational analysis

Nonmodular Architectures of Cognitive Systems based on Active Inference

Contact Info

Product

Resources

About