Models that learn how humans learn: The case of decision-making and its disorders

Abstract

Popular computational models of decision-making make specific assumptions about learning processes that may cause them to underfit observed behaviours. Here we suggest an alternative method using recurrent neural networks (RNNs) to generate a flexible family of models that have sufficient capacity to represent the complex learning and decision- making strategies used by humans. In this approach, an RNN is trained to predict the next action that a subject will take in a decision-making task and, in this way, learns to imitate the processes underlying subjects’ choices and their learning abilities. We demonstrate the benefits of this approach using a new dataset drawn from patients with either unipolar (n = 34) or bipolar (n = 33) depression and matched healthy controls (n = 34) making decisions on a two-armed bandit task. The results indicate that this new approach is better than baseline reinforcement-learning methods in terms of overall performance and its capacity to predict subjects’ choices. We show that the model can be interpreted using off-policy simulations and thereby provides a novel clustering of subjects’ learning processes—something that often eludes traditional approaches to modelling and behavioural analysis.

Using Deep Learning to Predict Human Decisions, and Cognitive Models to Explain Deep Learning Models

“…Different tools and approaches are being developed for this purpose, for example using visualisation to make linear regression models easy and quick to understand, and matching decision tree models to provide a systematic description of the model's behaviour [39][40][41][42]. In cognitive neuroscience, another approach to this problem is to use behavioural experimental tools to explain the model's behaviour [6,10]. One way to carry out this task is by examining the different experimental settings that make the model fail, known as adversarial examples, [24], which has a long tradition in cognitive psychology, from the use of visual illusions to study perception to the characterisation of biases in decision making [43].…”

“…However, as these models make specific assumptions about human behaviour and motivations, they may fall short if people’s behaviour is carried out in a completely different manner. An alternative approach is to use high capacity data-driven models not constraint by prior assumptions, which has the potential of breaking new ground in cognitive psychology [6,7]. However, relying on data driven models entails a different problem – the explainability or interpretability problem [8,9].…”

“…DNNs are becoming a popular tool in cognitive neuroscience [11]. DNNs have the capacity for implementing and representing complex computational steps, allowing them to reach human-like performance level in some tasks (and super human in others) without specifying an explicit description of the process [6,7,12]. This can be done by training a DNN model to perform a specific task, specifying the criterion for successful task performance, usually in the terms of accuracy or achievement of a goal, defined by a loss function.…”

Learning the structure of the world: The adaptive nature of state-space and action representations in multi-stage decision-making

“…In particular, although a wide range of models with different parameters were considered here, there were some differences between the pattern of stage 1 choices in the data shown in Fig 3(e) and the simulations of the best model shown in Fig 3(i). One way to address this issue is using recurrent neural networks (RNNs) instead of the RL family, which are more flexible and able to learn the details of the behavioural processes without relying on manually engineering the models [23]. Another limitation of the current work is that, although we provided evidence for the expansion of the state-space of the task, we did not provide any computational account for ‘how’ the states-space is acquired by the animals.…”

A neural algorithm for Drosophila linear and nonlinear decision-making

“…The proposed model has greater accuracy and is faster than traditional reinforcement learning method. Dezfouli et al 17 proposed a recurrent neural network to generate a flexible family of models that have sufficient capacity to represent the complex learning and decision-making strategies used by humans. On two-armed bandit task, the proposed method is better than baseline reinforcement-learning methods in terms of overall performance and its capacity to predict subjects’ choices.…”