Why Most Studies of Individual Differences With Inhibition Tasks Are Bound To Fail

Abstract

Establishing correlations among common inhibition tasks such as Stroop or flanker tasks has been proven quite difficult despite many attempts. It remains unknown whether this difficulty occurs because inhibition is a disparate set of phenomena or whether the analytical techiques to uncover a unified inhibition phenomenon fail in real-world contexts. In this paper, we explore the field-wide inability to assess whether inhibition is unified or disparate. We do so by showing that ordinary methods of correlating performance including those with latent variable models are doomed to fail because of trial noise (or, as it is sometimes called, measurement error). We then develop hierarchical models that account for variation across trials, variation across individuals, and covariation across individuals and tasks. These hierarchical models also fail to uncover correlations in typical designs for the same reasons. While we can charaterize the degree of trial noise, we cannot recover correlations in typical designs that enroll hundreds of people. We discuss possible improvements to study designs to help uncovering correlations, though we are not sure how feasible they are.

Processing speed, working memory, and executive functions: Independent or inter-related predictors of general intelligence

“…One very recently proposed solution to this may be to account for trial-to-trial noise by adopting a hierarchical modeling approach . However, there is also first evidence that this approach does not solve the measurement issues of difference measures (Rouder, Kumar, & Haaf, 2019), and thus does not provide an increment above the here used method of latent difference scores that are virtually error free (Kievit et al, 2018;Steyer et al, 1997).…”

Are There Reliable Qualitative Individual Difference in Cognition?

“…True effects are those that occur if we had an unlimited number of trials per person per condition. Yet, we only observe data from a limited number of trials, and the observed effects are perturbed from true effects by sample noise.Fortunately, we can make an educated guess about the effects of this noise as follows: Rouder et al (2019b) report that the trial-level standard deviation in Stroop and similar tasks is about 200 ms. Therefore, if we had 100 trials per person in a condition, we could reasonable expect to know an individuals' true mean RT in one condition with a standard error of 20 ms.…”

Religious Belief and Cognitive Conflict Sensitivity: A Preregistered fMRI Study

“…This might reflect that religiosity is unrelated to low-level cognitive control processes. At the same time, the null finding may also reflect the paradox that highly robust experimental effects -such as the Stroop effect-are often difficult to relate to reliable individual differences, irrespective of the specific individual difference construct of interest (Hedge, Powell, & Sumner, 2018;Rouder, Kumar, & Haaf, 2019). That is, because these effects are very robust and automatic ("everybody Stroops"), the between-subjects variability is by definition relatively small.…”

“…For correlational designs, this 'problem' of small between-subjects variability is further complicated by the presence of measurement error. Rouder et al (2019) demonstrated that the ratio of true variability (i.e., true differences between individuals) to trial noise (i.e., measurement error) is 1 : 7. This unfavorable ratio renders the mission to uncover individual differences in cognitive tasks difficult, if not even impossible.…”

“…In the current study, we did apply hierarchical modeling for the response time models, as well as the neural ACC models (incorporated in the first-level fMRI models in FSL and by adding the variance parameter of the beta's in the statistical models). Nevertheless, as acknowledged by Rouder et al (2019), characterizing the degree of measurement error does not imply that the real underlying individual differences can be recovered. This casts doubt on the feasibility to detect true individual variation in cognitive control tasks, and hence to uncover associations with other measures.…”