Latent goal models for dynamic strategic interaction

Abstract

Understanding the principles by which agents interact with both complex environments and each other is a key goal of decision neuroscience. However, most previous studies have used experimental paradigms in which choices are discrete (and few), play is static, and optimal solutions are known. Yet in natural environments, interactions between agents typically involve continuous action spaces, ongoing dynamics, and no known optimal solution. Here, we seek to bridge this divide by using a “penalty shot” task in which pairs of monkeys competed against each other in a competitive, real-time video game. We modeled monkeys’ strategies as driven by stochastically evolving goals, onscreen positions that served as set points for a control model that produced observed joystick movements. We fit this goal-based dynamical system model using approximate Bayesian inference methods, using neural networks to parameterize players’ goals as a dynamic mixture of Gaussian components. Our model is conceptually simple, constructed of interpretable components, and capable of generating synthetic data that capture the complexity of real player dynamics. We further characterized players’ strategies using the number of change points on each trial. We found that this complexity varied more across sessions than within sessions, and that more complex strategies benefited offensive players but not defensive players. Together, our experimental paradigm and model offer a powerful combination of tools for the study of realistic social dynamics in the laboratory setting.

Continuous decisions

“…As a result, single-trial analyses will likely be crucial (e.g. [99,100], also [71,72,76]). And these methods often benefit from denser recording techniques such as calcium imaging [101] and multi-contact multi-electrode recording ( [102,103] and [104,105]).…”

“…Just as importantly, while control systems are often assumed to be autonomous, with fixed, pre-specified set points, RL can be extended to the case of hierarchical learning and control [64][65][66][67][68][69][70]. That is, the RL framework can easily accommodate the idea of switching between policies or changing goals within the same policy [71,72]. These high-level changes are most often slower (in the case of goals) or sparser (for policy switches) and map neatly onto the experience of rarer deliberative decisions setting in motion automatic behaviours.…”

“…(a) Example: penalty kick task A recent study from our group illustrates this sort of integration of continuous control with sparse strategy changes. In the penalty kick task, monkeys and humans played a competitive video game against conspecifics that required continuous joystick input ( [71,72], figure 4). The game is based on the idea of a penalty shot in hockey.…”

The Neural Basis of Predictive Pursuit

“…Our work is directly inspired by important studies identifying mechanisms underlying pursuit in other animals 39, 40, 45 . Our work goes beyond these studies by developing a generative model, that is, a model that seeks to understand how the data are generated 46 . One benefit of the generative model is that it lets us probe how the decision is made at every time step and make guesses about the underlying mental process leading to decision.…”

Naturalistic decision-making: continuous, open-world, and recursive

“…Two recent paradigms from our groups illustrate precisely this sort of integration of continuous control with sparse strategy changes. In the first, monkeys and humans played a competitive video game against conspecifics that required continuous joystick input (McDonald et al, 2019;Iqbal et al, 2019, Figure 4). Based on the idea of a penalty shot in hockey, one player (the shooter) controlled the motion of a small circle (the puck), while the other (the goalie) controlled the vertical motion of a bar placed at the opposite side of the screen.…”