Putting bandits into context: How function learning supports decision making

Abstract

We introduce the contextual multi-armed bandit task as a framework to investigate learning and decision making in uncertain environments. In this novel paradigm, participants repeatedly choose between multiple options in order to maximise their rewards. The options are described by a number of contextual features which are predictive of the rewards through initially unknown functions. From their experience with choosing options and observing the consequences of their decisions, participants can learn about the functional relation between contexts and rewards and improve their decision strategy over time. In three experiments, we explore participants' behaviour in such learning environments. We predict participants' behaviour by context-blind (mean-tracking, Kalman filter) and contextual (Gaussian process and linear regression) learning approaches combined with different choice strategies. Participants are mostly able to learn about the context-reward functions and their behaviour is best described by a Gaussian process learning strategy which generalizes previous experience to similar instances. In a relatively simple task with binary features, they seem to combine this learning with a "probability of improvement" decision strategy which focuses on alternatives that are expected to lead to an improvement upon a current favourite option. In a task with continuous features that are linearly related to the rewards, participants seem to more explicitly balance exploration and exploitation. Finally, in a difficult learning environment where the relation between features and rewards is non-linear, some participants are again well-described by a Gaussian process learning strategy, whereas others revert to context-blind strategies.

Generalization and Search in Risky Environments

“…In order to satisfy this constraint, you should only explore options that—while uncertain—are likely to be “safe.” Such restricted exploration–exploitation problems are indeed common in daily life, from choosing which restaurant to visit (avoid food poisoning), where to buy a second‐hand car (avoid buying a lemon), to finding the shortest route home (avoid dangerous terrain). In our previous research on human behavior in contextual (Schulz, Konstantinidis, & Speekenbrink, ) and spatially correlated multi‐armed bandits (Wu, Schulz, Speekenbrink, Nelson, & Meder, in press), we found that human behavior in the search for rewards without constraints can be robustly described by a combination of a universal function learning mechanism and a decision strategy which explicitly balances an option’s expected reward and its attached uncertainty. The function learning mechanism was formalized as Gaussian process regression, which is a form of non‐parametric Bayesian regression that adapts its complexity to the data at hand (Griffiths, Lucas, Williams, & Kalish, ; Rasmussen, ), while the decision strategy was formalized as upper confidence bound sampling strategy (UCB; Auer, ).…”

“…Gaussian process regression is a non‐parametric Bayesian approach toward function learning which can perform generalization by making inductive inferences about unobserved outcomes. In past research we found that Gaussian process regression captures the inductive biases of human participants in a variety of explicit function learning tasks (Schulz, Tenenbaum, Duvenaud, Speekenbrink, & Gershman, ) and provides an accurate description of human generalization in contextual and spatially correlated multi‐armed bandits without the presence of unsafe outcomes (Schulz et al., ; Wu et al., in press).…”

“…Given the posterior mean and its attached standard error , we calculate the acquisition function of the upper confidence bound as where the exploration factor β determines how much reduction of uncertainty is valued (relative to exploiting known high‐value options) and is estimated as a free parameter. This sampling strategy has recently been found to describe human behavior well across different function exploration–exploitation tasks without additional constraints (Schulz et al., ; Wu et al., in press). Additionally, it has known performance guarantees in function optimization scenarios (Srinivas, Krause, Kakade, & Seeger, ).…”

Searching for rewards like a child means less generalization and more directed exploration

“…Rather than explaining development as a change in how we explore given some beliefs about the world, generalization-based accounts attribute developmental differences to the way we form our beliefs in the first place. Many studies have shown that human learners use structured knowledge about the environment to guide exploration (E. Schulz, Konstantinidis, & Speekenbrink, 2017), where the quality of these representations and the way that people utilize them to generalize across experiences can have a crucial impact on search behavior. Thus, development of more complex cognitive processes (Blanco et al, 2016), leading to broader generalizations, could also account for the observed developmental differences in sampling behavior.…”

Exploring the space of human exploration

“…As a first step toward modeling behavior in a probabilistic framework, we use a model that values both maximizing the probability of a correct query and a curiosity bonus, similar to recent work on human reinforcement learning (Schulz, Konstantinidis, & Speekenbrink, 2018;Wu et al, 2018). The curiosity bonus can be defined as information gain in the space of possible hypotheses (hidden codes).…”