Better Than Maximum Likelihood Estimation of Model-Based And Model-Free Learning Style

Abstract

Various decision-making systems work together to shape human behavior. Habitual and goal-directed systems are the two most important ones that are studied by reinforcement learning (RL), using model-free and model-based learning methods, respectively. Human behavior resembles the weighted combination of these two systems. Such a combination is modeled by the weighted sum of action values of the model-based and model-free systems. The weighting parameter has been mostly extracted by "maximum likelihood" or "maximum a-posteriori" estimation methods. In this study, we show these two well-known methods bring many challenges, and their respective extracted values are less reliable, especially in the case of limited sample size or at the proximity of extremes values. We propose that using k‑nearest neighbor, as a free format estimate, can improve the estimation error. k-nn uses global information extracted from the behavior such as stay probability, along with fitted values. The proposed method is examined by simulated experiments, where obtained results indicate the advantage of our method in reducing both bias and variance of the error. Investigation of the human behavior data from previous studies shows that the proposed method results in more statistically robust estimates in predicting other behavioral indices such as the number of gaze directions toward each target or symptoms of some psychiatric disorders. In brief, the proposed method increases the reliability of the estimated parameters and enhances the applicability of reinforcement learning paradigms in clinical trials.

Memory Trace Imbalance in Reinforcement and Punishment Systems Can Reinforce Implicit Choices Leading to Obsessive-Compulsive Behavior

“…We created 200 simulation data points using a model with the following parameters: α , 0.1 ± 0.05 (mean ± SD); β , 1 ± 0.2; ν + and ν − , randomly selected within 0.01–0.95. Parameter estimation was quite accurate (Yazdani et al, 2018) (Supplementary Figure 1). Specifically, Pearson’s r and the mean absolute error between the true and estimated ν + or ν − were 0.99 and 0.03, respectively (Supplementary Figure 1).…”