Signatures of heuristic-based directed exploration in two-step sequential decision task behaviour

Abstract

Processes formalized in classic Reinforcement Learning (RL) theory, such as model-based (MB) control and exploration strategies have proven fertile in cognitive and computational neuroscience, as well as computational psychiatry. Dysregulations in MB control and exploration and their neurocomputational underpinnings play a key role across several psychiatric disorders. Yet, computational accounts mostly study these processes in isolation. The current study extended standard hybrid models of a widely-used sequential RL-task (two-step task; TST) employed to measure MB control. We implemented and compared different computational model extensions for this task to quantify potential exploration mechanisms. In two independent data sets spanning two different variants of the task, an extension of a classical hybrid RL model with a heuristic-based exploration mechanism provided the best fit, and revealed a robust positive effect of directed exploration on choice probabilities in stage one of the task. Posterior predictive checks further showed that the extended model reproduced choice patterns present in both data sets. Results are discussed with respect to implications for computational psychiatry and the search for neurocognitive endophenotypes. Keywords: computational psychiatry, model-based, exploration, two-step task, neurocomputational endophenotypes