Inferring reward prediction errors in patients with schizophrenia: a dynamic reward task for reinforcement learning

Abstract

Abnormalities in the dopamine system have long been implicated in explanations of reinforcement learning and psychosis. The updated reward prediction error (RPE)—a discrepancy between the predicted and actual rewards—is thought to be encoded by dopaminergic neurons. Dysregulation of dopamine systems could alter the appraisal of stimuli and eventually lead to schizophrenia. Accordingly, the measurement of RPE provides a potential behavioral index for the evaluation of brain dopamine activity and psychotic symptoms. Here, we assess two features potentially crucial to the RPE process, namely belief formation and belief perseveration, via a probability learning task and reinforcement-learning modeling. Forty-five patients with schizophrenia [26 high-psychosis and 19 low-psychosis, based on their p1 and p3 scores in the positive-symptom subscales of the Positive and Negative Syndrome Scale (PANSS)] and 24 controls were tested in a feedback-based dynamic reward task for their RPE-related decision making. While task scores across the three groups were similar, matching law analysis revealed that the reward sensitivities of both psychosis groups were lower than that of controls. Trial-by-trial data were further fit with a reinforcement learning model using the Bayesian estimation approach. Model fitting results indicated that both psychosis groups tend to update their reward values more rapidly than controls. Moreover, among the three groups, high-psychosis patients had the lowest degree of choice perseveration. Lumping patients' data together, we also found that patients' perseveration appears to be negatively correlated (p = 0.09, trending toward significance) with their PANSS p1 + p3 scores. Our method provides an alternative for investigating reward-related learning and decision making in basic and clinical settings.

Cognitive training improves the disturbed behavioral architecture of schizophrenia-like rats, “Wisket”

“…The perseverative behavior (repetition of explorations) was also generally lower in the Wisket animals, which might be due to their lower level of exploratory activity in general. While several studies demonstrated increased perseveration in schizophrenic patients [27][28][29], our data are in agreement with recent human data, which showed decreased repetitive behavior in patients characterized by a dominance of positive psychotic symptoms [30]. As the method is based on the rats' natural tendency to explore their environment, factors such as lack of general exploratory drive and/or the ability to initiate actions (which relate closer to the negative symptoms of schizophrenia than to the cognitive deficits) could affect the investigation of the side-boxes.…”

“…The fact that from about the seventh day Wisket rats consumed the same amount of rewards as controls may reflect that the learning impairment was greatly overcome by the repetition of the tasks. Since cognitive training also improves cognitive performance in schizophrenic patients, and it can decrease the chance of acute psychiatric admission, too [13][14][15][16][17][18]30], we suggest the predictive validity of this model in this respect. Several brain structures (e.g.…”

Misfortune may be a blessing in disguise: Fairness perception and emotion modulate decision making

“…Reinforcement learning models (Glimcher, 2011;Niv, 2009;Sutton & Barto, 1998) have been used to investigate the process underlying trial-by-trial choices in feedback-based probabilistic learning tasks (e.g., Ahn, Busemeyer, Wagenmakers, & Stout, 2008;Ahn et al, 2014;Li et al, 2014;Liu et al, 2015;Rutledge et al, 2009). These models enable an assessment of the degree to which the participant updates his/her belief in response to feedback.…”

“…A Greenhouse-Geisser adjustment of degrees of freedom and a Bonferroni correction were used when necessary. To evaluate the strength of the evidence for the differences in the parameters between any two groups, following Fridberg et al (2010) and Li et al (2014), we computed the difference in values of the two posterior distributions in each run obtained from the Bayesian method for each parameter FIGU RE 2 Graphic display of the hybrid reinforcement learning model, in which the nodes represent the variables of interest and the arrows indicate the dependencies among the variables and checked whether the difference distribution was away from zero.…”

“…We developed an innovative version of the dictator game (Kahneman, Knetsch, & Thaler, 1986), which was modified from the ultimatum game (G€ uth, Schmittberger, & Schwarze, 1982), to induce emotional experiences in participants via the equality of a monetary assignment by the dictator. The monetary gain was subsequently used as the initial asset in a probabilistic gambling task (Li, Lai, Liu, & Hsu, 2014;Liu et al, 2015). In addition to conventional behavioral statistics, a reinforcement learning model (Glimcher, 2011;Niv, 2009;Sutton & Barto, 1998) using a hierarchical Bayesian estimation approach was adopted to fit the participants' trial-by-trial behavioral data to assess their latent process involved in the task, such as the sensitivity to loss aversion, the updating speed upon feedback, and the consistency of making decisions based on expected utility.…”