Learning the structure of the world: The adaptive nature of state-space and action representations in multi-stage decision-making

Abstract

State-space and action representations form the building blocks of decision-making processes in the brain; states map external cues to the current situation of the agent whereas actions provide the set of motor commands from which the agent can choose to achieve specific goals. Although these factors differ across environments, it is currently unknown whether or how accurately state and action representations are acquired by the agent because previous experiments have typically provided this information a priori through instruction or pre-training. Here we studied how state and action representations adapt to reflect the structure of the world when such a priori knowledge is not available. We used a sequential decision-making task in rats in which they were required to pass through multiple states before reaching the goal, and for which the number of states and how they map onto external cues were unknown a priori. We found that, early in training, animals selected actions as if the task was not sequential and outcomes were the immediate consequence of the most proximal action. During the course of training, however, rats recovered the true structure of the environment and made decisions based on the expanded state-space, reflecting the multiple stages of the task. Similarly, we found that the set of actions expanded with training, although the emergence of new action sequences was sensitive to the experimental parameters and specifics of the training procedure. We conclude that the profile of choices shows a gradual shift from simple representations to more complex structures compatible with the structure of the world. Author summaryEveryday decision-making tasks typically require taking multiple actions and passing through multiple states before reaching desired goals. Such states constitute the statespace of the task. Here we show that, contrary to current assumptions, the state-space is not static but rather expands during training as subjects discover new states that help them efficiently solve the task. Similarly, within the same task, we show that subjects initially only consider taking simple actions, but as training progresses the set of actions can PLOS Computational Biology | https://doi.The funders had no role in study design, data collection expand to include useful action sequences that reach the goal directly by passing through multiple states. These results provide evidence that state-space and action representations are not static but are acquired and then adapted to reflect the structure of the world.Two stage decision-making in rats PLOS Computational Biology | https://doi.org/10.

Hierarchical Action Control: Adaptive Collaboration Between Actions and Habits

“…Panels (A-D,G,H) are taken directly from Balleine (2013, 2019). Panels (E,F) are redrawn from Dezfouli and Balleine (2019).…”

Instrumental and Pavlovian Mechanisms in Alcohol Use Disorder

“…In light of these issues, alternative computational architectures have been proposed [for a review of other taxonomies, see 16] that might better align with classical behavioral findings across species. Furthermore, the recent back-translation of the two-step task for rodents [33][34][35][36][37] could also shed light on these issues. Indeed, translation of models and methodologies between preclinical and clinical research is crucial in the study of basic learning mechanisms in AUD.…”

“…This task has been recently back-translated to animal research, showing a similar behavioral pattern in rodents [33][34][35]. Moreover, a recent study has shown that rodents initially only use outcomes to drive behavior, but recover the structure of the environment over the course of learning and also use it to make decisions [36]. Indeed, some studies suggest that both rodents and humans can display predominantly model-based behavior following overtraining in the two-step task [37,38].…”

Behavioural Plasticity Can Help Evolving Agents in Dynamic Environments but at the Cost of Volatility

“…Many applications of neuroevolution focus on evolving the connection weights of ANNs [5,9,13,33], however, more complex approaches that evolve both the weights and topologies of ANNs exist [39,46]. Learning complex, sequential or multi-stage tasks is often hard for these neural controllers, as complete information about the environment-including the available actions, their cues, and their consequences-is not usually accessible [12,32]; this is also evident when environments are shared, as the actions of individuals change the context of the environment for others [5]. In nature, behavioural plasticity can be achieved with neuromodulation-a biological process whereby chemical signals are regulated (often also termed "modulated" or "gated") in the brain depending on environmental stimuli [1].…”

“…ANNs are just one example of an agent controller in which behaviour can be learned; we use ANNs in line with previous River Crossing testbeds [5,9,33] to explore how ANNs make decisions in social environments to solve tasks of variable complexity. Here, we define a multi-stage task as one that an agent must learn, and pass, through multiple states, and perform different behaviours in different contexts to achieve their goal; this definition is inspired by Reference [12].…”