Unpacking the exploration–exploitation tradeoff: A synthesis of human and animal literatures.

Abstract

Many decisions in the lives of animals and humans require a fine balance between the exploration of different options and the exploitation of their rewards. Do you buy the advertised car, or do you testdrive different models? Do you continue feeding from the current patch of flowers, or do you fly off to another one? Do you marry your current partner, or try your luck with someone else? The balance required in these situations is commonly referred to as the exploration-exploitation tradeoff. It features prominently in a wide range of research traditions, including learning, foraging, and decisionmaking literatures. Here, we integrate findings from these and other often-isolated literatures in order to gain a better understanding of the possible tradeoffs between exploration and exploitation, and we propose new theoretical insights that might guide future research. Specifically, we explore how potential tradeoffs depend on (1) the conceptualization of exploration and exploitation; (2) the influencing environmental, social, and individual factors; (3) the scale at which exploration and exploitation are considered; (4) the relationship and types of transitions between the two behaviors; and (5) the goals of the decision maker. We conclude that exploration and exploitation are best conceptualized as points on a continuum, and that the extent to which an agent's behavior can be interpreted as exploratory or exploitative depends upon the level of abstraction at which it is considered.

Magnitude and incentives: revisiting the overweighting of extreme events in risky decisions from experience

“…One important consideration pertinent to the aforementioned accounts is that they assume that participants have a complete and perfect representation of each option's outcomes and associated probabilities (in other words, perfect learning and memory). However, this is hardly the case as previous research in decisions from experience has shown that choice behavior is moderated by recency effects (e.g., Hertwig et al, 2004), sequential dependencies (e.g., Plonsky, Teodorescu, & Erev, 2015), exploration versus exploitation (e.g., Mehlhorn et al, 2015), and the search for predictable outcome patterns (e.g., Ashby et al, 2017;Shanks, Tunney, & McCarthy, 2002). To that end, future work would benefit from the inclusion of computational modeling approaches which can provide insight into the interplay between memory biases, diminishing sensitivity, recency effects, and the possible differential weighting of extreme outcomes and extreme probabilities.…”

Deconstructing the human algorithms for exploration

“…How humans resolve this dilemma has long puzzled psychologists and neuroscientists (Cohen, McClure, & Yu, 2007; Mehlhorn et al, 2015). Because the optimal solution is computationally intractable, humans must employ approximations or heuristics.…”

Putting bandits into context: How function learning supports decision making.

“…Email: e.schulz@cs.ucl.ac.uk. exploitation dilemma (e.g., Cohen, McClure, & Yu, 2007;Laureiro-Martínez, Brusoni, & Zollo, 2010;Mehlhorn et al, 2015): should you exploit your current but incomplete knowledge to pick an option you think is best, or should you explore something new and improve upon your knowledge in order to make better decisions in the future? While exploration is risky, in this case it is not blind.…”