Better than maximum likelihood estimation of model-based and model-free learning style

Abstract

Multiple decision making systems work together to shape the final choices in human behavior.Habitual and goal-directed systems are the two most important systems that are studied in the reinforcement learning (RL) literature by model-free and model-based learning methods. Human behavior resembles the weighted combination of these systems and such a combination is modeled by weighted summation of action`s value from the model based and model free systems. Extraction of this weighted parameter, which is important for many applications and computational modeling, has been mostly based on the maximum likelihood or maximum a posteriori methods. We show these methods bring many challenges and their respective extracted values are less reliable especially in the proximity of extremes values. We propose that using a free format learning method (k-nearest neighbor) which uses more information besides the fitted values e.g. global information like stay probability instead of trial by trial information can ameliorate the estimation error. The proposed method is examined by simulation and results show the advantage of the proposed method. In addition, investigation of the human behavior data from previous researchers proved the proposed method to result in more statistically robust results in predicting other behavioral indices such as the number of gaze directions toward each target. In brief, the proposed method increases the reliability of the estimated parameters and enhances the applicability of reinforcement learning paradigms in clinical trials.

Memory Trace Imbalance in Reinforcement and Punishment Systems Can Reinforce Implicit Choices Leading to Obsessive-Compulsive Behavior

“…We created 200 simulation data points using a model with the following parameters: α , 0.1 ± 0.05 (mean ± SD); β , 1 ± 0.2; ν + and ν − , randomly selected within 0.01–0.95. Parameter estimation was quite accurate 39 (Supplementary Figure 1). Specifically, Pearson’s r and the mean absolute error between the true and estimated ν + or ν − were 0.99 and 0.03, respectively (Supplementary Figure 1).…”

“…We created 200 simulation data points using a model with the following parameters: 𝛼, 0.1  0.05 (mean  SD); 𝛽, 1  0.2; 𝜈 + and 𝜈 − , randomly selected within 0.01-0.95. Parameter estimation was quite accurate 39 (Supplementary Figure 1).…”