Optimistic reinforcement learning: computational and neural bases

Abstract

While forming and updating beliefs about future life outcomes, people tend to consider good news and to disregard bad news. This tendency is supposed to support the optimism bias. Whether learning bias is specific to "high-level" abstract belief update or a particular expression of a more general "low-level" reinforcement learning process is unknown. Here we report evidence in favor of the second hypothesis. In a simple instrumental learning task, participants incorporated worse-than-expected outcomes at a lower rate compared to better-than-expected ones. This asymmetry was correlated across subjects with standard measure of dispositional optimism. Multimodal imaging indicated that inter-individual variability in the expression of asymmetric update relies on the dorsomedial prefrontal cortex at both morphological and functional levels. Our results constitute a new step in the understanding of the genesis of optimism bias at the neurocomputational level.

Optimistic update bias holds firm: Three tests of robustness following Shah et al.

“…The current set of results strongly support a valence dependent asymmetry in how participants update their beliefs, consistent with a large body of fast growing research (Eil and Rao, 2011, Garrett and Sharot, 2014, Garrett et al, 2014, Korn et al, 2012, Krieger et al, 2016, Kuzmanovic et al, 2015, Kuzmanovic et al, 2016, Lefebvre et al, 2016, Ma et al, 2016, Möbius et al, 2012, Sharot, 2011, Sharot and Garrett, 2016, Sharot et al, 2011, Sharot, Guitart-Masip et al, 2012). By pitting this asymmetry against three robustness tests suggested by critics of optimism (Shah et al, 2016) we find that it survives each of these tests, suggesting it is a pervasive phenomenon.…”

“…Numerous studies spanning behavioural economics (Eil and Rao, 2011, Krieger et al, 2016, Krieger et al, 2014, Möbius et al, 2012), psychology (Garrett and Sharot, 2014, Kuzmanovic et al, 2015, Moutsiana et al, 2013) and neuroscience (Garrett et al, 2014, Korn et al, 2012, Kuzmanovic et al, 2016, Lefebvre et al, 2016, Ma et al, 2016, Moutsiana et al, 2015, Sharot, Guitart-Masip et al, 2012, Sharot et al, 2011, Sharot, Kanai et al, 2012) have shown that people alter their beliefs to a greater extent in response to good news than bad news. This asymmetry can lead to a positive bias in beliefs regarding oneself, referred to as the superiority illusion (Hoorens, 1993, Kruger and Dunning, 1999, Svenson, 1981), and in beliefs regarding one’s future, referred to as unrealistic optimism (Calderon, 1993, Radcliffe and Klein, 2002, Shepperd et al, 2005, Weinstein, 1980, for review see Sharot & Garrett, 2016).…”

Valence biases factual and counterfactual learning in opposite directions

“…In addition to reduced reversal learning, and in accordance with a previous study (10), another behavioural feature that distinguished higher and lower bias participants was the preferred response rate in the Symmetric condition. In the Symmetric condition, both cues had the same reward probabilities (50%), such that there was no intrinsic "correct" response, allowing us to calculate a "preferred" response rate for each participant (defined as the choice rate of the option most frequently selected by a given participant, i.e.…”

“…We submitted the preferred choice rate to an ANOVA with experiment (1 or 2) and bias level (high and low) as between-subjects factors. The ANOVA showed a significant main effect of bias level higher biases were associated with an increased tendency to develop a preferred choice, even in the absence of a "correct" option, which naturally emerges from overweighting positive outcomes (10).…”

“…Despite the fact that standard models, built around the notion of computational and statistical optimality, prescribe that an agent should learn equally well from positive and negative obtained outcomes (7-9), previous studies have consistently shown that humans display a significant valence-induced bias. The bias generally goes in the direction of preferential learning from positive, compared to negative outcome prediction errors (10)(11)(12)(13)(14). This asymmetry in the effects of valence on RL could represent a "low-level" counterpart of the "good news/bad news" effect observed for "high-level" real life prospects, which has been suggested to contribute to maintaining an optimism bias(15).However, human RL cannot be reduced simply to learning from obtained outcomes.…”