Modeling the value of strategic actions in the Superior Colliculus

Abstract

In learning models of strategic game play, an agent constructs a valuation (action value) over possible future choices as a function of past actions and rewards. Choices are then stochastic functions of these action values. Our goal is to uncover a neural signal that correlates with the action value posited by behavioral learning models. We measured activity from neurons in the superior colliculus (SC), a midbrain region involved in planning saccadic eye movements, while monkeys performed two saccade tasks. In the strategic task, monkeys competed against a computer in a saccade version of the mixed-strategy game ”matching-pennies”. In the instructed task, saccades were elicited through explicit instruction rather than free choices. In both tasks neuronal activity and behavior were shaped by past actions and rewards with more recent events exerting a larger influence. Further, SC activity predicted upcoming choices during the strategic task and upcoming reaction times during the instructed task. Finally, we found that neuronal activity in both tasks correlated with an established learning model, the Experience Weighted Attraction model of action valuation (Camerer and Ho, 1999). Collectively, our results provide evidence that action values hypothesized by learning models are represented in the motor planning regions of the brain in a manner that could be used to select strategic actions.

The (Neural) Dynamics of Stochastic Choice

“…In particular, there is recognition that modeling binary choice as resulting from a single accumulator to dual threshold arises from mathematical convenience, not from neurobiological plausibility (Kiani et al 2014). The neurobiological evidence suggests that multiple, competing accumulators Z(t) are the primitive objects underlying the neural processes of decision (Mazurek et al 2003, Bogacz et al 2007, Beck et al 2008, Furman and Wang 2008, Thevarajah et al 2010. In particular, the evidence is incompatible with a stopping rule that depends on the difference between accumulators, and instead points to a fixed threshold that is Management Science, Articles in Advance, pp.…”

“…in this domain. One example is from a strategic decisionmaking task reported in Thevarajah et al (2010). In this experiment, a monkey subject played the matching pennies game against an opponent.…”

Cortical Signals for Rewarded Actions and Strategic Exploration

“…Consistent with this possibility, the animals indeed often displayed a small but significant bias to choose the same target rewarded in the previous trial, and to switch to the other target otherwise (Figure 2A; Lee et al, 2004; Seo and Lee, 2007). Referred to as a win-stay lose-switch (WSLS) strategy, this is naturally adopted by animals engaged in a wide variety of two alternative forced choice tasks (Sugrue et al, 2004; Dorris and Glimcher, 2004; Lau and Glimcher, 2005; Thevarajah et al, 2010). The WSLS strategy is suboptimal in the matching pennies task since it is exploited by the computer opponent.…”

The Relationship between Saccadic Choice and Reaction Times with Manipulations of Target Value

“…An added benefit of SRTs is that, in addition to providing an aggregate measure of value for a given prospect, their variability may provide insight into how subjective value is dynamically updated with trial by trial experience (Thevarajah et al, 2010). Indeed our preliminary analyses suggest trial by trial SRTs in single-target trials closely track trial by trial estimates of action value derived from reinforcement learning models (Milstein et al, 2010).…”