Does the reliability of computational models truly improve with hierarchical modeling? Some recommendations and considerations for the assessment of model parameter reliability

Abstract

Computational modeling of behavior is becoming a standard methodology in psychology, cognitive neuroscience, and computational psychiatry. In this approach, the reliability of the parameter estimates is an important issue. The most commonly used metric to assess it is the test-retest reliability, which quantifies the consistency of the parameter estimates obtained with multiple measurements. The reliability could be increased by improving the parameter estimation methods as well as the design of behavioral tasks. Studies using hierarchical (Bayesian) models have reported significant improvements in test-retest reliability. Hierarchical models assume prior distributions for individual parameter estimates. In this report, we point out that the test-retest reliability of parameter estimates based on prior distributions are strongly affected by heterogeneity in the sample population, possibly leading to overestimation of the reliability. When test-retest reliability is increased using prior distributions, it may indicate the presence of heterogeneity in the population and the need for subgrouping.