A psychometrics of individual differences in experimental tasks

Abstract

In modern individual-difference studies, researchers often correlate performance on various tasks to uncover common latent processes. Yet, in some sense, the results have been disappointing as correlations among tasks that seemingly have processes in common are often low. A pressing question then is whether these attenuated correlations reflect statistical considerations, such as a lack of individual variability on tasks, or substantive considerations, such as that inhibition in different tasks is not a unified concept. One problem in addressing this question is that researchers aggregate performance across trials to tally individual-by-task scores. It is tempting to think that aggregation is fine and that everything comes out in the wash. But as shown here, this aggregation may greatly attenuate measures of effect size and correlation. We propose an alternative analysis of task performance that is based on accounting for trial-by-trial variability along with the covariation of individuals' performance across tasks. The implementation is through common hierarchical models, and this treatment rescues classical concepts of effect size, reliability, and correlation for studying individual differences with experimental tasks. Using recent data from Hedge et al. Behavioral Research Methods, 50(3), 1166-1186, 2018 we show that there is Bayes-factor support for a lack of correlation between the Stroop and flanker task. This support for a lack of correlation indicates a psychologically relevant result-Stroop and flanker inhibition are seemingly unrelated, contradicting unified concepts of inhibition. Keywords Individual differences • Inhibition • Reliability • Hierarchical models • Bayesian inference In individual-differences studies, a number of variables are measured for each individual. The goal is to decompose the covariation among these variables into a lower-dimensional, theoretically relevant structure (Bollen, 1989; Skrondal & Rabe-Hesketh, 2004). Critical in this endeavor is understanding the psychometric properties of the measurements. Broadly speaking, variables used in individual-difference studies come from the following three classes: The first is the class of rather natural and easy-to-measure variables such age, weight, and gender. The second is the class of instruments such as personality and psychopathology instruments. Instruments have a fixed battery of questions and a fixed scoring algorithm. Most instruments have been benchmarked, and their reliability has been well established. The final class of variables is performance on experimental tasks. These experimental tasks are often used to assess cognitive abilities in memory, attention, and perception.

To pool or not to pool: Can we ignore cross-trial variability in FMRI?

“…3). In fact, our observations are more aligned with a similar assessment of test-retest reliability in psychometrics (Rouder and Haaf, 2019).…”

“…For example, Clark (1973) pointed out that neglecting the problem of the "language-as-fixed-effect fallacy ... can lead to serious error", and even alluded to earlier warnings that were largely ignored in the literature, including the report by Coleman (1964). Given the common practice of aggregation across trials, even classical experiments such as Stroop and flanker tasks, which one would anticipate to show high reliability, yielded lackluster results (Rouder and Haaf, 2019).…”

A chronometric model of the relationship between frontal midline theta functional connectivity and human intelligence

“…These cognitive processes are likely to reflect some of the most elementary components of the concept of cognitive control, but many more tasks that tap into overlapping and different executive processes are available. As there is currently no consensus regarding the best measures of cognitive control (Draheim, Tsukahara, et al, 2019;Hedge et al, 2018;Paap & Sawi, 2016;Rey-Mermet et al, 2018;Rouder & Haaf, 2019;Schubert & Rey-Mermet, 2019), it would be important to demonstrate that the association between functional connectivity and fluid intelligence found in the present study can be generalized to other measures of cognitive control. One of the most established measures of cognitive control is the antisaccade task, in which participants have to inhibit a prepotent saccade response towards a lateralized cue and make a voluntary saccade to the opposite side to identify a briefly presented target stimulus (Draheim, Tsukahara, et al, 2019;Kane, Bleckley, Conway, & Engle, 2001;Rey-Mermet et al, 2018).…”

“…Difference score measures, on the other hand, are supposed to reflect individual differences in cognitive control more validly because they isolate the cognitive control process of interest by directly measuring interindividual differences in intraindividual experimental effects (but see Rey-Mermet et al, 2019;and Schubert, Hagemann, Voss, Schankin, & Bergmann, 2015 for criticisms of the underlying assumption of additive factors that experimental manipulations selectively affect a single process of interest). However, experimentally validated difference score measures of cognitive control are often task-specific, show low reliabilities, and show little variation between individuals (Gärtner & Strobel, 2019;Hedge et al, 2018;Rey-Mermet et al, 2018;Rouder & Haaf, 2019). For these reasons, correlations between difference score measures of cognitive control and fluid intelligence are typically lower than those of mean performance measures and often fail to reach statistical significance at all (e.g., Friedman et al, 2006;Frischkorn et al, 2019;Rey-Mermet et al, 2019).…”

“…Taken together, these limitations call into question whether cognitive control can be reliably and validly measured using accuracy rates and mean reaction times in experimental tasks. Various solutions for improving the measurement of cognitive control have been suggested, including hierarchical modeling (Rouder & Haaf, 2019), the use of mathematical models , the introduction of individually calibrated response deadlines , and the development of new experimental tasks (Draheim, Tsukahara, Martin, Mashburn, & Engle, 2019). Undoubtedly, these promising approaches will allow a better evaluation of cognitive control as a psychometric construct and refine our understanding of the role of cognitive control processes in intelligence.…”