Mapping the unknown: The spatially correlated multi-armed bandit

Abstract

We introduce the spatially correlated multi-armed bandit as a task coupling function learning with the explorationexploitation trade-off. Participants interacted with bi-variate reward functions on a two-dimensional grid, with the goal of either gaining the largest average score or finding the largest payoff. By providing an opportunity to learn the underlying reward function through spatial correlations, we model to what extent people form beliefs about unexplored payoffs and how that guides search behavior. Participants adapted to assigned payoff conditions, performed better in smooth than in rough environments, and-surprisingly-sometimes performed equally well in short as in long search horizons. Our modeling results indicate a preference for local search options, which when accounted for, still suggests participants were best-described as forming local inferences about unexplored regions, combined with a search strategy that directly traded off between exploiting high expected rewards and exploring to reduce uncertainty about the spatial structure of rewards.

Active Function Learning

“…Moreover, they sampled inputs to learn about both the overall shape of the function as well as the location of high outputs. Interestingly, this result mirrors recent evidence that participants solve the explorationexploitation dilemma in reinforcement learning problems in a similar fashion (Wu et al, 2017), thereby hinting at the possibility of a universal sampling strategy underlying both information search and the search for rewards. Participants may not easily be able to turn off the exploitation part of their sampling strategy as they normally encounter a mix of exploration and exploitation problems in real life.…”

“…Moreover, GP models exhibit an inherent duality which makes them both a rule-based and a similarity-based model of function learning (see Lucas et al, 2015). As GP models have also been extended to account for exploratory behavior in function optimization tasks (Wu, Schulz, Speekenbrink, Nelson, & Meder, 2017), we will utilize them as candidate active learning models here.…”

“…UCB sampling will, on average, converge to both high knowledge about the underlying function and sampling the highest possible outcomes. It has recently been found to describe human behavior well in a function exploration-exploitation paradigm (Wu et al, 2017). We also combined the linear regression model with two additional sampling strategies.…”

Connecting conceptual and spatial search via a model of generalization

“…Thus, we presented information about both spatial and conceptual features in both search tasks, but only one of them was relevant for generalization and predicting rewards. At the beginning of each round only a single randomly chosen option was revealed (i.e., displayed the numerical reward and corresponding color aid), whereby subjects had a limited horizon of 10 actions in each round (40% of the total search space; similar to Wu et al, 2017), thereby inducing an exploration-exploitation trade-off.…”

Finding structure in multi-armed bandits

“…As the true generative model of the reward distributions on each round is a Gaussian Process (GP), we can also model structured function learning using the same framework (Rasmussen & Williams, 2006;. Gaussian Process regression is a nonparametric Bayesian method that has been successfully applied as a model of how people generalize in contextual (Schulz, Konstantinidis, & Speekenbrink, 2017) and spatially correlated (Wu, Schulz, Speekenbrink, Nelson, & Meder, 2017) multi-armed bandits.…”