Evidence of systematic attenuation in the measurement of cognitive deficits in schizophrenia.

Abstract

Cognitive tasks that are too hard or too easy produce imprecise measurements of ability, which, in turn, attenuates group differences and can lead to inaccurate conclusions in clinical research. We aimed to illustrate this problem using a popular experimental measure of working memory—the N-back task—and to suggest corrective strategies for measuring working memory and other cognitive deficits in schizophrenia. Samples of undergraduates (n = 42), community controls (n = 25), outpatients with schizophrenia (n = 33), and inpatients with schizophrenia (n = 17) completed the N-back. Predictors of task difficulty—including load, number of word syllables, and presentation time—were experimentally manipulated. Using a methodology that combined techniques from signal detection theory and item response theory, we examined predictors of difficulty and precision on the N-back task. Load and item type were the two strongest predictors of difficulty. Measurement precision was associated with ability, and ability varied by group; as a result, patients were measured more precisely than controls. Although difficulty was well matched to the ability levels of impaired examinees, most task conditions were too easy for non-impaired participants. In a simulation study, N-back tasks primarily consisting of 1- and 2-back load conditions were unreliable, and attenuated effect size (Cohen’s d) by as much as 50%. The results suggest that N-back tasks, as commonly designed, may underestimate patients’ cognitive deficits due to non-optimized measurement properties. Overall, this cautionary study provides a template for identifying and correcting measurement problems in clinical studies of abnormal cognition.

A signal detection–item response theory model for evaluating neuropsychological measures

“…A well-known axiom of psychometric theory demonstrates that associations between variables are attenuated to the extent that measures of those variables are unreliable (Haynes et al, 2011; Spearman, 1904). For example, measurement error attenuates group differences in test performance between samples of cognitively impaired versus healthy examinees (Thomas et al, 2017). Moreover, the risks of interpreting differential deficits based on measures of distinct cognitive abilities with unequal reliabilities have been well documented (e.g., Chapman & Chapman, 1973).…”

Add to cart: an investigation of the efficiency of social comparisons to thin-ideal images in the context of online shopping

“…yel-low). Increasing the number of syllables that need to be rehearsed likely increases cognitive load (Mano et al, 2013;Thomas et al, 2017). Additionally, evidence shows that the capacity of visual working memory which would be activated when remembering colours, is only about four-items (Alvarez & Cavanagh, 2004).…”

Latent Variable Modeling and Adaptive Testing for Experimental Cognitive Psychopathology Research

“…Measurement error is directly tied to discriminating power, in that as measurement error increases, discrimination decreases. When discriminating power differs between tests and groups, differential deficits can be artificially inflated or attenuated (Chapman & Chapman, 1973; Thomas et al, 2017).…”

“…Moreover, because reliability is a function of measurement error, and because effect size is a function of reliability, ability–difficulty mismatch results in poor reliability and weakened effect size. Thus, experimental investigators who use factorial or parametric manipulations to alter a task’s difficulty will also systematically alter the reliability of a task’s measures of individual differences and in clinical applications will alter the size of the between group effect (Thomas et al, 2017).…”