The curse of optimism: a persistent distraction by novelty

Abstract

Human curiosity has been interpreted as a drive for exploration and modeled by intrinsically motivated reinforcement learning algorithms. An unresolved challenge in machine learning is that these algorithms are prone to distraction by reward-independent stochastic stimuli. We ask whether humans exhibit the same distraction pattern in their behavior as the algorithms. To answer this question, we design a multi-step decision-making paradigm containing an unknown number of states in a stochastic part of the environment, combined with a reward-manipulation that varies optimism of participants regarding the availability of rewards. We show that (i) participants exhibit a persistent distraction by novelty in the stochastic part; (ii) reward optimism increases this distraction; and (iii) the persistent distraction is consistent with the predictions of algorithms driven by novelty but not with ‘optimal’ algorithms driven by information-gain. Our results suggest that humans use sup-optimal but computationally cheap curiosity-driven policies for exploration in complex environments.

Representational similarity modulates neural and behavioral signatures of novelty

“…We developed a generalized model of novelty computation that leverages kernel mixture models to capture the effect of stimulus similarities on novelty computation and extends current computational novelty models to continuous state spaces. In tasks where all stimuli are discrete and distinct, our kernel-based novelty model is equivalent to count-based novelty [14,15,[18][19][20][21][22][23]46]. We show that kernel-based novelty captures novelty responses in mouse V1 [39] that are unexplained by count-based models.…”

“…Most algorithmic models of novelty computation in the brain are based on estimating the counts [14, 18, 20, 22, 23] or the count-based frequency [15, 19, 21] at which stimuli (e.g., sensory stimuli, spatial locations etc.) have been observed: the more often or frequently a stimulus is observed, the less novel it is.…”

“…Novelty signaling in the brain is known to facilitate learning [1][2][3][4][5][6][7], enhance sensory processing [8][9][10][11] and drive behaviour [12][13][14][15][16][17] in humans and animals encountering unfamiliar stimuli or situations. Computational models of novelty are valuable tools to explain and predict the effects of novelty signaling on neural activity and behaviour [14,15,[18][19][20][21]. Existing computational novelty models often compute novelty as a decreasing function of the number of times a stimulus has been encountered previously ('count-based' novelty [18,20,22,23], see [24] for a review).…”

“…Some variations of these models consider only binary counts [14, 18], i.e. whether a stimulus is completely unfamiliar or not, while others base novelty computation on the stimulus frequency [15, 19, 21], i.e. the relative number of stimulus observations.…”