The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single rewardprediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversallearning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With highresolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.

Curiosity drives information seeking and promotes learning. Prior work has focused on how curiosity is elicited by intrinsic qualities of information, leaving open questions about how curiosity, exploration, and learning are shaped by the environment. Here we examine how temporal dynamics of the learning environment shape curiosity and learning. Participants (n = 71) foraged for the answer to trivia questions in two conditions that differed only in their temporal statistics. In one condition, the timing of information delivery followed a uniform distribution, while in another it followed a heavy-tailed distribution. We found that the two conditions elicited distinct responses in both behavior and pupil dilation: participants were more likely to wait for information and to later remember it in the uniform distribution. By contrast, participants showed greater surprise, evidenced in a spike in pupil dilation, when presented with the answers in the heavy-tailed distribution. Furthermore, pupil dilation was inversely related to curiosity and memory, suggesting that temporal uncertainty may interfere with the positive effects of curiosity on learning. Our findings demonstrate that the predicted timing of information delivery influences information seeking, memory, and physiological arousal, suggesting that information is best learned when it is both intrinsically interesting and presented within a temporally predictable environment.

Curiosity is a powerful determinant of behavior. The past decade has seen a surge of scientific research on curiosity, an endeavor recently imbibed with urgency by the WHO, which set managing information-seeking as a public health goal during pandemics. And yet, a fundamental aspect of curiosity has remained unresolved: its relationship to utility. Is curiosity a drive towards information simply for the sake of obtaining that information, or is it a rational drive towards optimal learning? We leveraged people’s curiosity about COVID-19 to study information-seeking and learning in a large sample (n=5376) during the spring of 2020. Our findings reveal that curiosity is goal-rational in that it maximizes the personal utility of learning. Personal utility, unlike normative economic utility, is contingent on a person’s motivational state. On the basis of these findings, we explain information-seeking during the pandemic with a rational theoretical framework for curiosity.

No abstract