Biased belief updating and suboptimal choice in foraging decisions

Abstract

In many choice scenarios, including prey, employment, and mate search, options are not encountered simultaneously and so cannot be directly compared. Deciding which ones optimally to engage, and which to forego, requires developing accurate beliefs about the overall distribution of prospects. However, the role of learning in this process -and how biases due to learning may affect choice -are poorly understood. In three experiments, we adapted a classic prey selection task from foraging theory to examine how individuals kept track of an environment's reward rate and adjusted their choices in response to its fluctuations. In accord with qualitative predictions from optimal foraging models, participants adjusted their selectivity to the richness of the environment: becoming less selective in poorer environments and increasing acceptance of less profitable options. These preference shifts were observed not just in response to global (between block) manipulations of the offer distributions, but also to local, trial-by-trial offer variation within a block, suggesting an incremental learning rule. Further offering evidence into the learning process, these preference changes were more pronounced when the environment improved compared to when it deteriorated. All these observations were best explained by a trial-by-trial learning model in which participants estimate the overall reward rate, but with upward vs. downward changes controlled by separate learning rates. A failure to adjust expectations sufficiently when an environment becomes worse leads to suboptimal choices: options that are valuable given the environmental conditions are rejected in the false expectation that better options will materialize. These findings offer a previously unappreciated parallel in the serial choice setting of observations of asymmetric updating and resulting biased (often overoptimistic) estimates in other domains.Previously, we have shown (Constantino and Daw, 2015) that when individuals undertake a different type of foraging problem ("patch leaving"), their choices are well explained by an errordriven incremental learning rule for estimating the environment's reward rate (Schultz et al., 1997;Sutton and Barto, 1998), which then serves as a comparator for determining which prospects are acceptable. However, systematic deviations from optimal thresholds were detected, which are even more pronounced in participants under stress (Lenow et al., 2017) or with depleted levels of dopamine . This suggests that there are circumstances under which individuals may misestimate the environment's rate of return, and in doing so make choices out of step with what the MVT would predict. However, in the line of studies on patch foraging, these biases have simply been treated as fixed choice tendencies, and in particular have not been shown to arise from the underlying learning process.

Sympathetic and parasympathetic involvement in time constrained sequential foraging

“…Replicating past findings (Garrett and Daw, 2019), the Asymmetric Model again provided a superior fit (see Table 1) to the choice data than the Symmetric Model ( (19) = 3.14, = 0.005, paired sample ttests comparing Leave One Out cross validation scores for the Asymmetry versus the Symmetric Model) with information integration again being biased in a positive direction ( p > M : = 1.80, < 0.05 one-tailed). Prediction errors that caused to shift upwards (following receipt of a reward) had a greater impact than prediction errors that caused to shift downwards (following the absence of a reward).…”

“…Our first behavioral analysis attempted to replicate the Garrett and Daw (2019) finding regarding asymmetric belief updating. To this end, we ran a three-way mixed ANOVA of mean capture-rate as a function of between-group factor (RP, PR) and two repeated-measures factors (boom, downturn) and (hi, intermediate, low).…”

“…Appraising sequential offers of reward relative to an unknown future opportunity and a time cost requires an optimization policy that draws on a belief about the richness characteristics of the current environment. Across a range of experiments, including reinforcement-learning tasks, belief updating paradigms and prey selection, information integration shows a positive bias Sharot, 2014, 2017;Garrett et al, , 2018Kuzmanovic et al, 2015Kuzmanovic et al, , 2016Eil and Rao, 2011;Korn et al, 2012;Kuzmanovic and Rigoux, 2017;Lefebvre et al, 2017;Garrett and Daw, 2019).…”

“…In each model, was initialized at the beginning of the experiment to the arithmetic average reward rate across the experiment, but subsequently carried over between environments (Constantino and Daw, 2015;Garrett and Daw, 2019). For each participant, we estimated the free parameters of the To formally test for differences in learning rates ( p , M ) we estimated the covariance matrix Σ x over the group level parameters using the Hessian of the model likelihood (Oakes, 1999) and then used a contrast c z Σ x c to compute the standard error on the difference p -M .…”

“…Recent work in humans further resolves the computational challenge of learning dynamic environmental richness, by demonstrating that sequential choice behavior is best captured by an MVT inspired learning model. Specifically, both decisions to leave a patch and explore the environment in patch foraging (Constantino and Daw, 2015;Lenow et al, 2017) and capture behavior in prey selection (Garrett and Daw, 2019) adhere to the MVT predicted optimality policy that compares yields against fluctuating environmental richness which is learned via a standard delta rule. This later work further demonstrated that beliefs about environmental richness update with asymmetric bias, whereby improvements are learned at a higher rate than deteriorations; the 'naïve perseverance of optimism' (Garrett and Daw, 2019).…”