Influences of Reinforcement and Choice Histories on Choice Behavior in Actor-Critic Learning

Abstract

Reinforcement learning models have been used in many studies in the fields of neuroscience and psychology to model choice behavior and underlying computational processes. Models based on action values, which represent the expected reward from actions (e.g., Q-learning model), have been commonly used for this purpose. Meanwhile, the actor-critic learning model, in which the policy update and evaluation of an expected reward for a given state are performed in separate systems (actor and critic, respectively), has attracted attention due to its ability to explain the characteristics of various behaviors of living systems. However, the statistical property of the model behavior (i.e., how the choice depends on past rewards and choices) remains elusive. In this study, we examine the history dependence of the actor-critic model based on theoretical considerations and numerical simulations while considering the similarities with and differences from Q-learning models. We show that in actor-critic learning, a specific interaction between past reward and choice, which differs from Q-learning, influences the current choice. We also show that actor-critic learning predicts qualitatively different behavior from Q-learning, as the higher the expectation is, the less likely the behavior will be chosen afterwards. This study provides useful information for inferring computational and psychological principles from behavior by clarifying how actor-critic learning manifests in choice behavior.

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

“…Like H t (a), its counterpart H t (s t ,a) can also be modeled with the accumulating hysteresis trace [21]. Along with the alternative of a replacing trace (see Methods), another more constrained implementation of hysteretic accumulation could be based on an action-prediction error (or choice-prediction error) with analogy to the reward-prediction error [40,[42][43][44][45][46][47]96,143,144,178,181]. The actionprediction error has been framed as "value-free", but this label and that of H t (s t ,a) as "habit strength" (cf.…”

“…That the effects illuminated herein are so parsimonious and demonstrably extractable means that comparable studies of RL and other sequential tasks generally stand to benefit from considering bias and hysteresis as part of due diligence-even if the main focus of inquiry is directed elsewhere. Being more representative of actual behavior, the expanded 5-parameter base model 0CE1 aims to enhance parameter identifiability with respect to actual RL as opposed to action-specific components of variance that may mimic or otherwise obscure signatures of learning with spurious correlations [17,18,27,28,[39][40][41][42][43][44][45][46][47]. Before making additional assumptions, parsimoniously imposing action-specific parameters with first priority can be beneficial as a sort of regularization for learning parameters that in practice are nontrivial to extract and estimate.…”

“…A more comprehensive model of action selection can also enhance identifiability with respect to actual learning (or lack thereof) as opposed to other components of variance that may mimic or otherwise obscure signatures of learning with spurious correlations across the finite sequence of actions [17,18,27,28,[39][40][41][42][43][44][45][46][47]. As external reinforcement promotes consistent repetition of responses within a state, so too can action bias, and both repetition and alternation from hysteresis can coincidentally align with the reward contingencies of the sequence of states.…”

Autonomous air traffic separation assurance through machine learning

“…In the policy gradient algorithm, the policy gradient updates the Actor based on actual rewards. Critic is responsible for learning reward mechanisms, and can discover potential rewards in the current state through the relationship between the learning environment and rewards, which is used to generate Actor actions [27,34,1]. Furthermore, Actor-Critic method adds evaluation network to policy gradient algorithm to fit Q value and the Actor is used to map the state to the action.…”