Optimal reward harvesting in complex perceptual environments

Abstract

The ability to choose rapidly among multiple targets embedded in a complex perceptual environment is key to survival. Targets may differ in their reward value as well as in their low-level perceptual properties (e.g., visual saliency). Previous studies investigated separately the impact of either value or saliency on choice; thus, it is not known how the brain combines these two variables during decision making. We addressed this question with three experiments in which human subjects attempted to maximize their monetary earnings by rapidly choosing items from a brief display. Each display contained several worthless items (distractors) as well as two targets, whose value and saliency were varied systematically. We compared the behavioral data with the predictions of three computational models assuming that (i) subjects seek the most valuable item in the display, (ii) subjects seek the most easily detectable item, and (iii) subjects behave as an ideal Bayesian observer who combines both factors to maximize the expected reward within each trial. Regardless of the type of motor response used to express the choices, we find that decisions are influenced by both value and feature-contrast in a way that is consistent with the ideal Bayesian observer, even when the targets' feature-contrast is varied unpredictably between trials. This suggests that individuals are able to harvest rewards optimally and dynamically under time pressure while seeking multiple targets embedded in perceptual clutter. A nimals and humans often need to make rapid choices among multiple targets embedded in a noisy perceptual environment. Consider, for example, a predator deciding which of several prey to pursue. The more valuable targets might be perceptually less salient, and thus harder to find (e.g., camouflaged prey), while less valuable targets may be perceptually more salient and easier to find. To solve this task, the animal needs to combine the perceptual and valuerelated information while making choices. This raises a fundamental question: Are rapid choices in cluttered environments dominated by value information (e.g., biased toward seeking the more valuable items) or by perceptual information (e.g., biased toward seeking the more easily and quickly detectable items)?Understanding how perceptual saliency and value information are combined to make decisions is important for several reasons. From a computational perspective, it is not known how the brain trades off saliency and value under time pressure, especially when they have opposing influences on the decision: Is it optimized for reward harvesting, or is it based on simpler principles of choosing the most valuable or the most easily detectable item? The brain's solution to this tradeoff is not obvious, because items that are more salient are easier to find (1, 2) and, other things being equal, have a higher probability of yielding a reward. From a behavioral perspective, it is not known whether humans take into account saliency-induced variations in probability when making decisi...

Value-driven attentional capture

“…Several recent studies have shown that the voluntary deployment of attention is influenced by reward (10)(11)(12)(13)(14)(15)(16)(17)(18)(19). In contrast to the ample evidence that voluntary deployment of attention to taskrelevant stimuli is affected by reward, the evidence for an influence of the value assigned to stimuli through reward learning on involuntary attentional capture is negative or equivocal (10)(11)(12)26).…”

“…Voluntary attention to stimuli that predict reward is an effective mechanism for efficiently selecting valuable stimuli (9). Many studies have shown that reward facilitates voluntary attention to task-relevant stimuli, and that reward-based strategies and priorities strongly influence attentional performance (10)(11)(12)(13)(14)(15)(16)(17)(18)(19).…”

How do magnitude and frequency of monetary reward guide visual search?

“…One possibility is that our visual search task is broadly insensitive not just to spatial reward magnitude learning but to many forms of magnitude learning. As mentioned in the introduction, several researchers have reported robust prioritization of features that are associated with greater monetary reward (e.g., Anderson, et al, 2011;Kiss et al, 2009;Navalpakkam et al, 2010). In Experiment 2, we test whether the visual search task used in Experiment 1 produces learning of feature magnitude.…”

“…Just one study has reported that individuals learn to bias spatial attention to locations containing more valuable targets . Note that, overall, the number of studies of attention and reward have surged in recent years, most of which have shown robust prioritization of rewarded nonspatial features (e.g., Anderson, Laurent, & Yantis, 2011;Della Libera & Chelazzi, 2006;Hickey, Chelazzi, & Theeuwes, 2010;Kiss, Driver, & Eimer, 2009;Navalpakkam, Koch, Rangel, & Perona, 2010). Given the traditionally strong focus of attention research on spatial attention (Bisley & Goldberg, 2010;Fecteau & Munoz, 2006), one might expect more published studies demonstrating learning of spatial reward magnitude.…”

Automaticity and cognitive control in the learned predictiveness effect.

“…Indeed, it is likely that value-driven and physical bottom-up processes operate independently. When visual salience and value information vary independently within a complex perceptual context, both variables combine to influence performance in a way that maximises reward (Navalpakkam, Koch, Rangel, & Perona, 2010).…”