What is the state space of the world for real animals?

Abstract

A key concept in reinforcement learning (RL) is that of a state space. A state space is an abstract representation of the world using which the statistical relations in the world can be described. The simplest form of RL, model free RL, is widely applied to explain animal behavior in numerous neuroscientific studies. More complex RL versions assume that animals build and store an explicit model of the world in memory. Inherent to the application of both these classes of RL to animal behavior are assumptions about the underlying state space formed by animals, especially in relation to their representation of time. Here, we explicitly list these assumptions and show that they have several problematic implications. We propose a solution for these problems by using a continuous time Markov renewal process model of the state space. We hope that our explicit treatment results in a serious consideration of these issues when applying RL models to real animals.

A Spectrum of Time Horizons for Dopamine Signals

“…This model is not designed to handle prolonged time horizons that might span multiple trials (15). Furthermore, the splitting of experience into discrete, equally-fine sub-states becomes ever more artificial as inter-trial intervals get larger and more variable (51, 52).…”

“…5A; (52)). Furthermore, the splitting of experience into discrete, equally-fine sub-states becomes ever more artificial as inter-trial intervals get longer and more variable (53, 54).…”

How do real animals account for the passage of time during associative learning?

“…Another formalism within the general framework of RL is a semi-Markov/Markov renewal process model that explicitly learns the distribution of time intervals between consecutive state transitions (Bradtke & Duff, 1994; Daw et al, 2006; Namboodiri, 2021). Though very similar, there is one difference between a semi-Markov and Markov renewal process-based state space.…”

“…As the value of states is tied to the currently active state in a semi-Markov model, the value is stationary for the duration of this entire state. However, in a Markov renewal process, the value function can be defined in continuous time, as the states and transition times are treated separately (Namboodiri, 2021). While these models avoid any issues with breaking up the flow of time into states, they nevertheless suffer from some limitations.…”

Relative salience signaling within a thalamo-orbitofrontal circuit governs learning rate

“…Importantly, the ITI was similarly split to 30 states (the maximum possible ITI was divided into 3 s states starting after the previous outcome state). We have previously noted that this commonly used state space makes problematic assumptions about real animals, 85 but nevertheless use it here for its simplicity in illustrating our main claim about learning rate. The task was simulated based on the temporal parameters listed above and the timeline for the task was converted to a timeline of states, with each time step lasting 3 s. The temporal difference value function and update were then applied on this state space.…”