Deconstructing the human algorithms for exploration

Abstract

The dilemma between information gathering (exploration) and reward seeking (exploitation) is a fundamental problem for reinforcement learning agents. How humans resolve this dilemma is still an open question, because experiments have provided equivocal evidence about the underlying algorithms used by humans. We show that two families of algorithms can be distinguished in terms of how uncertainty affects exploration. Algorithms based on uncertainty bonuses predict a change in response bias as a function of uncertainty, whereas algorithms based on sampling predict a change in response slope. Two experiments provide evidence for both bias and slope changes, and computational modeling confirms that a hybrid model is the best quantitative account of the data.

Temporal discounting correlates with directed exploration but not with random exploration

“…The fact that random exploration does not correlate with temporal discounting is also consistent with theories of random exploration (Watkins, 1989;Sutton & Barto, 2018). Moreover, this apparent dissociation between directed and random exploration is consistent with other findings showing that directed and random exploration have different computational properties ( (Gershman, 2018), different age dependence (Somerville et al, 2017), and may rely on dissociable neural systems (Zajkowski et al, 2017;Gershman & Tzovaras, 2018;Warren et al, 2017). In this regard it is notable that directed exploration appears to rely on the same frontal systems thought to underlie temporal discounting (Frank et al, 2009, Gershman & Tzovaras, 2018, Zajkowski et al, 2017Doya, 2002;McClure, Laibson, Loewenstein, & Cohen, 2004;McClure, Ericson, Laibson, Loewenstein, & Cohen, 2007), while random exploration does not.…”

Search as a simple take-the-best heuristic

“…How well does our full model perform compared to alternative modelling approaches? Because most models in the literature either focus on the exploration (e.g., [19,9,31]) or the exploitation mechanism (e.g., [10,11,20]), we tested these two main components separately. For this, we first kept the exploitation mechanism unchanged, and tested five different models for the exploration phase: 1) our take-the-best model, 2) a probabilistic variation of that model in which the individual chooses to rely on the payoff cue or the visibility cue based on the most rewarding neighbouring solution [32,33], 3) a typical hill-climbing model where exploration is always directed towards the most-rewarding adjacent position [34], 4) a "blind search" model in which the exploration is only guided by novelty, and 5) a random search model in which the next position is randomly chosen among the adjacent solutions.…”

“…While TTB constitutes a valid model to describe how people make decisions between two options [37], we have shown that it can also be used to describe search behaviours. Recent research on human search behaviours distinguishes between directed and undirected exploration [32,31,19]. In multi-armed bandit tasks undirected exploration refers to the stochasticity of the search process causing random exploration decisions.…”

“…In contrast directed exploration seeks out solutions that are informative about the underlying reward distribution [32]. Both components play a important role in solving exploration exploitation dilemma and are often considered as "two core components of exploration" [31]. Our full model stands along the same lines: the noise parameter accounts for undirected exploration, whereas the novelty cue guides the exploration towards unknown regions of the solution space and hence accounts for directed exploration.…”

Generalizing to generalize: when (and when not) to be compositional in task structure learning

“…where ω is a free parameter that controls the degree of uncertain-guided exploration (see Gershman, 2018, for a thorough discussion of uncertainty guided exploration). In our simulations, ω was sampled from the distribution logit −1 (ω) ∼ N (−1, 1).…”