Toward a principled Bayesian workflow in cognitive science.

Abstract

Experiments in research on memory, language, and in other areas of cognitive science are increasingly being analyzed using Bayesian methods. This has been facilitated by the development of probabilistic programming languages such as Stan, and easily accessible front-end packages such as brms. The utility of Bayesian methods, however, ultimately depends on the relevance of the Bayesian model, in particular whether or not it accurately captures the structure of the data and the data analyst's domain expertise. Even with powerful software, the analyst is responsible for verifying the utility of their model. To demonstrate this point, we introduce a principled Bayesian workflow (Betancourt, 2018) to cognitive science. Using a concrete working example, we describe basic questions one should ask about the model: prior predictive checks, computational faithfulness, model sensitivity, and posterior predictive checks. The running example for demonstrating the workflow is data on reading times with a linguistic manipulation of object versus subject relative clause sentences. This principled Bayesian workflow also demonstrates how to use domain knowledge to inform prior distributions. It provides guidelines and checks for valid data analysis, avoiding overfitting complex models to noise, and capturing relevant data structure in a probabilistic model. Given the increasing use of Bayesian methods, we aim to discuss how these methods can be properly employed to obtain robust answers to scientific questions. All data and code accompanying this article are available from https://osf.io/b2vx9/.

Pragmatic constraints do not prevent the co-activation of alternative names: evidence from sequential naming tasks with one and two speakers

“…We evaluated the evidence in favour of the interaction by computing Bayes factors for the contrast between a model containing the interaction and a model not containing it. We used weakly informative priors based on prior predictive checks (Schad, Betancourt, & Vasishth, 2019) for all parameters in the model and varied the prior for the critical interaction to examine the influence of this prior on the resulting Bayes Factor (Vasishth, Nicenboim, Beckman, Li, & Kong, 2018). Table 3 displays the Bayes Factors in favour of the null hypothesis (the model without the interaction) and the 95% credible interval for the interaction term (for each experiment and the crossexperimental analysis).…”

A Bayesian approach to dynamical modeling of eye-movement control in reading of normal, mirrored, and scrambled texts

“…Dynamical cognitive models represent a framework that permits the test of very specific hypotheses about cognitive processes underlying human behavior (Schütt et al, 2017), in particular when such models are investigated in a principled Bayesian workflow (Schad et al, 2019). A strong test of dynamical models, however, requires time-ordered observations, such as eye movements, brain imaging, or single-cell recordings or other types of high-density behavioral data.…”

“…In the following, several procedures are implemented to ensure computational faithfulness of model and sampling method, to evaluate the predictive power of the fitted model, and to make inferences to explain observed variability with assumed underlying model behavior. We adopted the principled Bayesian workflow discussed in Schad et al (2019) to secure validity and reliability of our numerical inferences. The steps taken are as follows:…”

“…In the present theoretical study, we fitted the SWIFT model to diverse reading patterns to evaluate whether the model can reproduce the variability between experimental conditions and baseline, following a principled workflow that improves model fit, inferences, and comparability (Schad et al, 2019). In this approach, the likelihood function plays a key role as an objective optimization target for model fitting that was introduced in an earlier publication by Seelig et al (2020).…”

Interference patterns in subject-verb agreement and reflexives revisited: A large-sample study

“…A further potential use of posterior predictive distributions is that they could be used as informative priors on a future replication attempt. For extensive discussion of prior and posterior predictive distributions for model evaluation in the context of cognitive science applications, see Schad, Betancourt, and Vasishth (2019).…”