The relative merit of empirical priors in non-identifiable and sloppy models: Applications to models of learning and decision-making

Abstract

Formal modeling approaches to cognition provide a principled characterization of observed responses in terms of a set of postulated processes, specifically in terms of parameters that modulate the latter. These model-based characterizations are useful to the extent that there is a clear, one-to-one mapping between parameters and model expectations (identifiability) and that parameters can be recovered from reasonably sized data using a typical experimental design (recoverability). These properties are sometimes not met for certain combinations of model classes and data. One suggestion to improve parameter identifiability and recoverability involves the use of "empirical priors", which constrain parameters according to a previously observed distribution of values. We assessed the efficacy of this proposal using a combination of real and artificial data. Our results showed that a point-estimate variant of the empirical-prior method could not improve parameter recovery systematically. We identified the source of poor parameter recovery in the low information content of the data. As a follow-up step, we developed a fully Bayesian variant of the empirical-prior method and assessed its performance. We find that even such a method that takes the covariance structure of the parameter distributions into account cannot reliably improve parameter recovery. We conclude that researchers should invest additional efforts in improving the informativeness of their experimental designs, as many of the problems associated to impoverished designs cannot be alleviated by modern statistical methods alone.

Theoretically Informed Generative Models Can Advance the Psychological and Brain Sciences: Lessons from the Reliability Paradox

“…Although we focused on test-retest reliability, there are many other properties worth exploring including parameter identifiability (e.g., Spektor & Kellen, 2018), parameter recovery (e.g., Ahn et al, 2011;Haines et al, 2018;Miletić, Turner et al, 2017), tests of selective influence (a form of construct validity where experimental manipulations cause expected changes in parameter values; Criss, 2010), and parameter convergence between behavioral models and models derived at other levels of analysis (e.g., with trait or neural models; Haines et al, 2020;Turner et al, 2017). Bayesian analysis facilitates joint estimation of all model parameters and their hypothesized relations, thus allowing for proper calibration of uncertainty in key parameters (e.g., test-retest reliability).…”

Response time models separate single- and dual-process accounts of memory-based decisions

“…To address question 2, we calculated the 95% HDIs of the posterior group-level parameter distributions. If the parameters are identifiable, the 95% HDI of should include the true data-generating parameter value (see also Spektor and Kellen, 2018). We observed good identifiability of the group-level parameters for both models (Figs.…”

Response time models separate single- and dual-process accounts of memory-based decisions

“…The DPDM suffered from stronger sloppiness, especially between To address question 2, we calculated the 95% HDIs of the posterior group-level parameter distributions. If the parameters are identifiable, the 95% HDI of should include the true data-generating parameter value (see also Spektor & Kellen, 2018). We observed good identifiability of the group-level parameters for both models (Figs.…”