The elusive effects of incidental anxiety on reinforcement-learning

Abstract

Anxiety is a common affective state, characterized by the subjectively unpleasant feelings of dread over an anticipated event. Anxiety is suspected to have important negative consequences on cognition, decision-making and learning. Yet, despite a recent surge in studies investigating the specific effects of anxiety on reinforcement-learning, no coherent picture has emerged. Here, we investigated the effects of incidental anxiety on instrumental reinforcement learning, while addressing several issues and defaults identified in a focused literature review. We used a rich experimental design, featuring both a learning and a transfer phase, and a manipulation of outcomes valence (gains vs losses). In two variants (N = 2x50) of this experimental paradigm, incidental anxiety was induced with an established threat-of-shock paradigm. Model-free results show that incidental anxiety effects seem limited to a small, but specific increase in post-learning performance measured by a transfer task. A comprehensive modelling effort revealed that, irrespective of the effects of anxiety, individuals give more weight to positive than negative outcomes, and tend to experience the omission of a loss as a gain (and vice versa). However, in line with results from our targeted literature survey, isolating specific computational effects of anxiety on learning per se proved to be challenging. Overall, our results suggest that learning mechanisms are more complex than traditionally presumed, and raise important concerns about the robustness of the effects of anxiety previously identified in simple reinforcement-learning studies.

The impact of incidental anxiety on the neural signature of mentalizing

“…A specific advantage of this fMRI design is that it allows us to induce anxiety for a prolonged period of time. Furthermore, this approach reduces task switching demands (see Engelmann et al, 2015;Ting et al, 2020). At the beginning of each block, participants were informed of the current condition via a block cue that indicated whether vignettes were presented in the economic games or life stories domain, whether questions would assess beliefs or outcomes, and whether shocks would be administered at unpredictable time points throughout the next block or not.…”

“…Anxiety is an emotion that is particularly prevalent for human behavior (Bandelow et al 2015), as its presence signals the occurrence of negative events that can be crucial for survival. Recent research has begun to investigate the effects of such incidental anxiety on cognitive processes including attention (Bar-Haim et al, 2007;Bradley et al, 2010;Cisler & Koster, 2010;MacLeod & Mathews, 1988), memory (Andreotti et al, 2011;Balderston et al, 2017;Bolton & Robinson, 2017;Vytal et al, 2013), and learning (Becker et al, 2019;Berghorst et al, 2013;Browning et al, 2015;Cavanagh et al, 2019;DeVido et al, 2009;Grupe, 2017;Robinson et al, 2013;Safra et al, 2018;Stevens et al, 2014;Ting et al, 2020). Despite this recent surge in the interest concerning the effects of incidental anxiety on cognitive processes and its neural correlates, relatively little is known about its impact on arguably one of the most important cognitive processes for human interaction, namely social cognitions such as mentalizing.…”

SalemGarcia_2021

“…In addition, and similar to (Ting et al, 2020b), we also tested different implementations of how the context value is learnt in the partial information contexts (see Methods). Overall, our model space included 10 models, which we checked were identifiable, and whose parameters were estimable (Methods and Supplementary Methods 2.2.2-2.2.3).…”

The shadowing effect of initial expectation on learning asymmetry

“…However, results for the learning asymmetry in more elementary forms of updates such as RL are rather mixed. While some studies using standard RL paradigms found that humans’ positive learning rates were larger than the negative ones, yielding an optimistic RL bias [ 4 , 16 , 21 ]. Other studies, however, obtained opposite results with larger negative learning rates [ 6 , 7 , 22 ], consistent with the prevalent psychological phenomenon “bad is stronger than good” [ 23 ].…”

“…However, a majority of recent studies focused exclusively on the role of learning rates in characterizing participants’ learning behavior and considered Q 0 as a mundane initialization parameter, due to the belief that the impact of initial expectations would be “washed out” after enough trials of learning. Indeed, while some studies initialized Q 0 to zero, probably reflecting the fact that participants possessed no information about options before the task [ 6 – 8 , 21 , 25 ]; other studies adopted the median or mean values of the possible option outcomes as Q 0 , corresponding to an a priori expectation of receiving different outcomes with equal probabilities [ 4 , 26 – 28 ]. Only a limited number of studies tested the models where Q 0 was treated as a free parameter [ 29 , 30 ].…”