Theoretically Informed Generative Models Can Advance the Psychological and Brain Sciences: Lessons from the Reliability Paradox

Abstract

Behavioral tasks (e.g., Stroop task) that produce replicable group-level effects (e.g., Stroop effect) often fail to reliably capture individual differences between participants (e.g., low test-retest reliability). This “reliability paradox” has led many researchers to conclude that most behavioral tasks cannot be used to develop and advance theories of individual differences. However, these conclusions are derived from statistical models that provide only superficial summary descriptions of behavioral data, thereby ignoring theoretically-relevant data-generating mechanisms that underly individual-level behavior. More generally, such descriptive methods lack the flexibility to test and develop increasingly complex theories of individual differences. To resolve this theory-description gap, we present generative modeling approaches, which involve using background knowledge to specify how behavior is generated at the individual level, and in turn how the distributions of individual-level mechanisms are characterized at the group level—all in a single joint model. Generative modeling shifts our focus away from estimating descriptive statistical “effects” toward estimating psychologically meaningful parameters, while simultaneously accounting for measurement error that would otherwise attenuate individual difference correlations. Using simulations and empirical data from the Implicit Association Test and Stroop, Flanker, Posner Cueing, and Delay Discounting tasks, we demonstrate how generative models yield (1) higher test-retest reliability estimates, and (2) more theoretically informative parameter estimates relative to traditional statistical approaches. Our results reclaim optimism regarding the utility of behavioral paradigms for testing and advancing theories of individual differences, and emphasize the importance of formally specifying and checking model assumptions to reduce theory-description gaps and facilitate principled theory development.

Trial and error: a hierarchical modeling approach to test-retest assessment

“…We added two more distributions, Student's t and exGaussian, because of their ability of handling skewed data as well as outliers. In addition, instead of assuming a diagonal matrix R (0) for the variancecovariance structure of the varying intercepts τ rs as in Haines (2020), we adopted a generic R (0) in the BML model (17), leading to largely consistent results with those reported by Haines et al (2020) with regard to the three common distributions. However, the added exGaussian outperformed all alternatives per the information criterion through leave-one-out cross-validation.…”

“…That is, the correlation ρ 0 embedded in the variance-covariance matrix R (0) for cross-subject varying intercepts τ rs captures the TRR for the average effect. It is worth noting that ρ 0 was assumed to be 0 in Haines et al (2020). First, the LME formulation (5) with a single condition effect can be considered as a special case of the LME model (12) -equivalent to a condition contrast with the effects from the two conditions being identical.…”

“…First, the LME formulation (5) with a single condition effect can be considered as a special case of the LME model (12) -equivalent to a condition contrast with the effects from the two conditions being identical. This equivalence would not be true for the model adopted by Haines et al (2020). Furthermore, the LME framework (12) is more generic and consist between both intercept and slope effects in the sense that TRR is assumed to exist for each of the two conditions, their contrast as well as their average effect.…”

“…In addition, based on recent foundational work Haines et al, 2020), we aim to 1) build a hierarchical model that more accurately characterizes test-retest data, which accounts for crosstrial variability and estimates TRR as a correlation;…”

Collecting big data with small screens: Group tests of children’s cognition with touchscreen tablets are reliable and valid

“…For example, even an "excellent" reliability of 90% would correspond to a 95% confidence interval of ± 9.3 points (assuming an IQ-like scale with a standard deviation of 15; Revelle & Condon, 2018). Further improvements to reliability could be possible by utilizing more sophisticated modeling techniques (Farrell & Lewandowsky, 2018;Haines et al, 2020), or adaptive testing procedures (Harrison, Collins, & Müllensiefen, 2017).…”