Paradoxical choice and the reinforcing value of information

Abstract

Signals that reduce uncertainty can be valuable because well-informed decision-makers can better align their preferences to opportunities. However, some birds and mammals display an appetite for informative signals that cannot be used to increase returns. We explore the role that reward-predictive stimuli have in fostering such preferences, aiming at distinguishing between two putative underlying mechanisms. The ‘information hypothesis’ proposes that reducing uncertainty is reinforcing per se, somewhat consistently with the concept of curiosity: a motivation to know in the absence of tractable extrinsic benefits. In contrast, the ‘conditioned reinforcement hypothesis’, an associative account, proposes asymmetries in secondarily acquired reinforcement: post-choice stimuli announcing forthcoming rewards (S+) reinforce responses more than stimuli signalling no rewards (S−) inhibit responses. In three treatments, rats faced two equally profitable options delivering food probabilistically after a fixed delay. In the informative option (Info), food or no food was signalled immediately after choice, whereas in the non-informative option (NoInfo) outcomes were uncertain until the delay lapsed. Subjects preferred Info when (1) both outcomes were explicitly signalled by salient auditory cues, (2) only forthcoming food delivery was explicitly signalled, and (3) only the absence of forthcoming reward was explicitly signalled. Acquisition was slower in (3), when food was not explicitly signalled, showing that signals for positive outcomes have a greater influence on the development of preference than signals for negative ones. Our results are consistent with an elaborated conditioned reinforcement account, and with the conjecture that both uncertainty reduction and conditioned reinforcement jointly act to generate preference.

Learning to Choose: Behavioral Dynamics Underlying the Initial Acquisition of Decision Making

“…This result suggests rats deliberate (i.e., show slowing on dual-offer trials compared to single-offer trials) and likely engage in some comparative process when deciding between options of known value. Our findings are quite different from the studies by the Kacelnik group that motivated our experiment (Fig 6 in Shapiro et al, 2008; Fig 2 in Ojeda et al, 2018; Fig 4 in Ajuwon et al, 2023). Their studies reported much longer overall response latencies (on the order of seconds).…”

Do goldfish like to be informed?

“…One possibility concerns the effects of conditioned reinforcement. One mechanistic explanation for learned preferences for advanced information argues that subjects select the informative option because 'good news' signals anticipating forthcoming rewards acquire reinforcing properties by association with outcomes of positive valence (8,19,28,69). If any putative inhibitory effects of signals for 'bad news' in the informative option are relatively small or ignored by subjects, overall, subjects would form stronger associations between choice and reward in the informative option, but these effects may be quantitatively different between species.…”

“…In Phase 1, when presented with S-, subjects swam away from the stimulus (Figure 3;(see also 43)), and in Phase 2 subjects rarely selected S-when it was presented alongside other stimuli (Supplementary Figure 3), suggesting that stimuli for reward omission can acquire aversive properties which could inhibit Info preference. Manipulations involving the removal of S-(e.g., 19,73,74) could help to elucidate the extent to which conditioned inhibition plays a role in goldfish preferences for advanced information. Subjects in our study learnt the task contingencies, showing appropriate responses to outcome-predictive stimuli, but they did not develop a preference for informative stimuli foretelling good and bad news equally often.…”

Representations of the intrinsic value of information in mouse orbitofrontal cortex

“…In our information seeking task, mice move from the center port to a side port where they experience a 10s delay before receiving a water reward. We expect that were we to shorten this delay, if mice value information because it grants them access to informative cues in advance of the delayed reward, the value of information should also be reduced because it either diminished the duration of pleasurable anticipation or aversive uncertainty 6,16,19,49 . We have therefore manipulated the subjective value of information in this task by varying the length of delay on both sides, Information and no information, across blocks of sessions.…”