Reliability Concerns in the Repeated Computerized Assessment of Attention in Children

Abstract

Assessment of attentional processes via computerized assessment is frequently used to quantify intra-individual cognitive improvement or decline in response to treatment. However, assessment of intra-individual change is highly dependent on sufficient test reliability. We examined the test–retest reliability of selected variables from one popular computerized continuous performance test (CPT)—i.e., the Conners’ CPT – Second Edition (CPT-II). Participants were 39 healthy children (20 girls) ages 6–18 without intellectual impairment (mean PPVT-III SS = 102.6), LD, or psychiatric disorders (DICA-IV). Test–retest reliability over the 3–8 month interval (mean = 6 months) was acceptable (Intraclass Correlations [ICC] = .82 to .92) on comparison measures (Beery Test of Visual Perception, WISC-IV Block Design, PPVT-III). In contrast, test–retest reliability was only modest for CPT-II raw scores (ICCs ranging from .62 to .82) and T-scores (ICCs ranging from .33 to .65) for variables of interest (Omissions, Commissions, Variability, Hit Reaction Time, and Attentiveness). Using test–retest reliability information published in the CPT-II manual, 90% confidence intervals based on reliable change index (RCI) methodology were constructed to examine the significance of test–retest difference/change scores. Of the participants in this sample of typically developing youth, 30% generated intra-individual changes in T-scores on the Omissions and Attentiveness variables that exceeded the 90% confidence intervals and qualified as “statistically rare” changes in score. These results suggest a considerable degree of normal variability in CPT-II test scores over extended test–retest intervals, and suggest a need for caution when interpreting test score changes in neurologically unstable clinical populations.

Response time variability and response inhibition predict affective problems in adolescent girls, not in boys: the TRAILS study

“…This is in line with previous studies that showed that response time variability in children and adolescents was of modest stability [81] and that response inhibition is stable in test–retest [85]. The present study shows that baseline response speed and working memory were only coincidentally associated with affective problems in the present study.…”

The Arizona Cognitive Test Battery for Down Syndrome: Test-Retest Reliability and Practice Effects

“…Taking into account past research on extensive reaction time variability in children (Zabel, von Thomsen, Cole, Martin, & Mahone, 2009), this finding was unexpected, but suggests that measures that are quite simple in their demands, such as reaction time, may allow for the consistent measurement of cognitive outcome in ID syndromes.…”

Behavioral Inhibition and Addiction

“…Reliability studies of continuous performance tasks typically involve Conners' version of the task, which is widely used in the assessment of ADHD (Conners, 2004). These tasks show high test-retest reliability (>0.85) of commission errors in child and adult samples over short time intervals, from 1 to 2 weeks (e.g., Halperin, Sharma, Greenblatt, and Schwartz, 1991;Soreni, Crosbie, Ickowicz, and Schachar, 2009), and modest reliabilities (0.62 to 0.82) over longer intervals from 3 to 8 months (e.g., Kaminski, Groff, and Glutting, 2009;Zabel et al, 2009). Other more complex versions, such as the immediate and delayed memory task, typically used in adults, also demonstrate high test-retest reliability across consecutive testing days .…”