Computational mechanisms underlying motivation to earn symbolic reinforcers

Abstract

Reinforcement learning (RL) is a theoretical framework that describes how agents learn to select options that maximize rewards and minimize punishments over time. We often make choices, however, to obtain symbolic reinforcers (e.g. money, points) that can later be exchanged for primary reinforcers (e.g. food, drink). Although symbolic reinforcers are motivating, little is understood about the neural or computational mechanisms underlying the motivation to earn them. In the present study, we examined how monkeys learn to make choices that maximize fluid rewards through reinforcement with tokens. The question addressed here is how the value of a state, which is a function of multiple task features (e.g. current number of accumulated tokens, choice options, task epoch, trials since last delivery of primary reinforcer, etc.), drives value and affects motivation. We constructed a Markov decision process model that computes the value of task states given task features to capture the motivational state of the animal. Fixation times, choice reaction times, and abort frequency were all significantly related to values of task states during the tokens task (n=5 monkeys). Furthermore, the model makes predictions for how neural responses could change on a moment-by-moment basis relative to changes in state value. Together, this task and model allow us to capture learning and behavior related to symbolic reinforcement.

Ventral frontostriatal circuitry mediates the computation of reinforcement from symbolic gains and losses

“…We found that OFC encoded accumulated tokens and changes in tokens at both the single-cell and population levels. These findings are consistent with previous work showing that OFC codes state information about the environment [37][38][39][40] , as tokens and token updates define states and state transitions in this task 41 . State representation is also referred to as a cognitive map, particularly when states have to be inferred 42,43 .…”