Model-free and model-based learning processes in the updating of explicit and implicit evaluations

Abstract

Evaluating stimuli along a good-bad dimension is a fundamental computation performed by the human mind. In recent decades, research has documented dissociations and associations between explicit (i.e., self-reported) and implicit (i.e., indirectly measured) forms of evaluations. However, it is unclear whether such dissociations arise from relatively more superficial differences in measurement techniques or from deeper differences in the processes by which explicit and implicit evaluations are acquired and represented. The present project (total N = 2,354) relies on the computationally well-specified distinction between model-based and model-free reinforcement learning to investigate the unique and shared aspects of explicit and implicit evaluations. Study 1 used a revaluation procedure to reveal that, whereas explicit evaluations of novel targets are updated via model-free and model-based processes, implicit evaluations depend on the former but are impervious to the latter. Studies 2 and 3 demonstrated the robustness of this effect to (i) the number of stimulus exposures in the revaluation phase and (ii) the deterministic vs. probabilistic nature of initial reinforcement. These findings provide a framework, going beyond traditional dual-process and single-process accounts, to highlight the context-sensitivity and long-term recalcitrance of implicit evaluations as well as variations in their relationship with their explicit counterparts. These results also suggest avenues for designing theoretically guided interventions to produce change in implicit evaluations.Implicit Association Test | implicit evaluations | implicit social cognition | model-free vs. model-based learning | reinforcement learning

The relationship between implicit intergroup attitudes and beliefs

“…Moreover, at least under some conditions, implicit beliefs seem to be more closely related to explicit attitudes than they are to explicit beliefs (42). Finally, a recent investigation using a reinforcement learning perspective has found that whereas implicit attitudes track a history of personally experienced rewards and punishments, explicit attitudes also reflect additional information about the structure of the environment (43). However, it should be noted that most of these results have been obtained using implicit measures of attitude and, as such, may not readily generalize to implicit measures of beliefs.…”

The roles of online and offline replay in planning

“…However, rather than 16 preplay immediately prior to choice, we found evidence of on-task replay following feedback 17 alone, suggesting a third potential factor impacting on the timing and content of replaythe 18 need to minimize memory load by embedding new information in ones' policies as soon as it 19 is received. 20 Critically, the timing and content of replay differed across individuals in a manner that links 21 with their dominant mode of planning. More model-based subjects tended to replay 22 trajectories during learning, predominantly reflecting choices they were likely to reconsider.…”

“…18 Subjects were first told explicitly how many reward points were associated with each of the 19 eight images. Subjects were then trained on these image-reward associations until they 20 reliably chose the more rewarding image of any presented pair (see Image-reward training). 21 Next, the rules of the state-space task were explained (see State-space task), and multiple- 22 choice questions were used to ensure that subjects understood these instructions.…”

“…Every six consecutive trials featured 6 different starting 18 locations. The one exception to this were the first of the 24 two-move trials of the 19 experiment, where in order to facilitate learning, each starting location repeated for two 20 consecutive trials (a similar measure was also implemented for one-move trials during 21 training; see State-space training). Subjects' performance improved substantially in the second of such pairs of trials (Δproportion of optimal first choices = +0.15, 95% CI = +0.11 23 to +0.18, p < 0.001, Bootstrap test).…”

“…The algorithm estimates expected outcomes by multiplying the number of points associated 18 with an image with the probability of transitioning to that image, integrating over all potential 19 future images: 20 ( ,1 , ) = ∑ ( ,1 , , ) ( ).…”