Exploration-exploitation mechanisms in recurrent neural networks and human learners in restless bandit problems

Abstract

A key feature of animal and human decision-making is to balance exploring unknown options for information gain (directed exploration) versus exploiting known options for immediate reward, which is often examined using restless bandit problems. Recurrent neural network models (RNNs) have recently gained traction in both human and systems neuroscience work on reinforcement learning. Here we comprehensively compared the performance of a range of RNN architectures as well as human learners on restless four-armed bandit problems. The best-performing architecture (LSTM network with computation noise) exhibited human-level performance. Cognitive modeling showed that human and RNN behavior is best described by a learning model with terms accounting for perseveration and directed exploration. However, whereas human learners exhibited a positive effect of uncertainty on choice probability (directed exploration), RNNs showed the reverse effect (uncertainty aversion), in conjunction with increased perseveration. RNN hidden unit dynamics revealed that exploratory choices were associated with a disruption of choice predictive signals during states of low state value, resembling a win-stay-loose-shift strategy, and resonating with previous single unit recording findings in monkey prefrontal cortex. During exploration trials, RNN selected exploration targets predominantly based on their recent value, but tended to avoid more uncertain options. Our results highlight both similarities and differences between exploration behavior as it emerges in RNNs, and computational mechanisms identified in cognitive and systems neuroscience work.

A meta reinforcement learning account of behavioral adaptation to volatility in recurrent neural networks

“…Future work is required to extend these findings to other tasks used to study learning rate adaptation (Behrens et al, 2007(Behrens et al, , 2008Browning et al, 2015;Cook et al, 2019;Gagne et al, 2020) (Wang et al, 2018). Finally, computational modeling was restricted to an RL model, that disregarded effects of higher-order choice perseveration (Lau & Glimcher, 2008;Miller et al, 2019;Tuzsus et al, 2024) and more complex directed (i.e. uncertainty-guided) exploration mechanisms (Chakroun et al, 2020;Wiehler et al, 2021;Wilson et al, 2014).…”

“…Several limitations need to be addressed. First, we focused on the effects of training regime on behavioral adaptation to volatility in one specific neural network architecture (a 48-unit LSTM network trained with the A2C algorithm (Mnih et al, 2016) and Weber noise (Findling & Wyart, 2020)), based on our previous results that this architecture exhibits human-level performance on four-armed restless bandit problems (Tuzsus et al, 2024). Previous work on meta-RL applied similar RNN architectures (Binz & Schulz, 2022;Blanco-Pozo et al, 2024;Findling & Wyart, 2020;Hattori et al, 2023;Molano-Mazón et al, 2023;Wang et al, 2017Wang et al, , 2018, but generally, conclusions are limited to the architectures examined.…”

“…Stochasticity estimates may also serve as a baseline for the detection of volatility (Soltani & Izquierdo, 2019), which could be tested directly in RNNs via different training regimes (Soltani & Izquierdo, 2019). Third, we focused on a restless bandit tasks used in many previous human and nonhuman studies (Chakroun et al, 2020;Daw et al, 2006;Ebitz et al, 2018;Speekenbrink & Konstantinidis, 2015;Wiehler et al, 2021) as well as modeling RNN studies (Findling & Wyart, 2020;Tuzsus et al, 2024;Wang et al, 2017). Future work is required to extend these findings to other tasks used to study learning rate adaptation (Behrens et al, 2007(Behrens et al, , 2008Browning et al, 2015;Cook et al, 2019;Gagne et al, 2020) (Wang et al, 2018).…”

“…For each training regime (see below) n=30 RNN models based on LSTM units endowed with computational noise (Findling & Wyart, 2020; Tuzsus et al, 2024) where trained. Standard LSTM dynamics (Hochreiter & Schmidhuber, 1997) were characterized by following equations: …”

“…(2) Computational noise following Weber's law of intensity sensation accounts for adaptive behavior and neural signals in human learners exposed to volatile RL tasks (Findling et al, 2019;Findling & Wyart, 2021). Importantly, such computational noise was also shown to improve adaptation of RNN behavior to adverse RL environments (e.g., tasks with increased volatility) (Findling & Wyart, 2020), which allows them to perform as good as human learners in the exact same volatile bandit task (Tuzsus et al, 2024).…”