Settling the Score: Can CPT-3 Embedded Validity Indicators Distinguish Between Credible and Non-Credible Responders Referred for ADHD and/or SLD?

Abstract

Objective: The purpose of the present study was to further investigate the clinical utility of individual and composite indicators within the CPT-3 as embedded validity indicators (EVIs) given the discrepant findings of previous investigations. Methods: A total of 201 adults undergoing psychoeducational evaluation for ADHD and/or Specific Learning Disorder (SLD) were divided into credible ( n = 159) and non-credible ( n = 42) groups based on five criterion measures. Results: Receiver operating characteristic curves (ROC) revealed that 5/9 individual indicators and 2/4 composite indicators met minimally acceptable classification accuracy of ≥0.70 (AUC = 0.43–0.78). Individual (0.16–0.45) and composite indicators (0.23–0.35) demonstrated low sensitivity when using cutoffs that maintained specificity ≥90%. Conclusion: Given the lack of stability across studies, further research is needed before recommending any specific cutoff be used in clinical practice with individuals seeking psychoeducational assessment.

Combining accuracy scores with time cutoffs improves the specificity of the Word Choice Test.

“…On one hand, this discrepancy serves as a reminder that classification accuracy derived from student samples tends to shift the signal detection profile toward an α bias (i.e., prioritizing the detection of noncredible responding over minimizing false positives), which is the opposite of the widely accepted convention in PVT research (i.e., optimizing for specificity; Chafetz, 2022). On the other hand, cutoffs developed in clinical patients may be unnecessarily conservative (and even counterproductive) in settings where genuine and severe cognitive deficits are unlikely, such as baseline testing for sports-related concussion management (Abeare et al, 2018; Abeare, Messa, et al, 2019; Lichtenstein & Merz, 2019), psychoeducational assessments (Bing-Canar et al, 2022; Harrison & Armstrong, 2022; Robinson et al, 2023), or academic research using healthy undergraduate students (An et al, 2017; Roye et al, 2019).…”

Utility of the D-KEFS color–word interference test as an embedded validity indicator in psychoeducational evaluations.

“…Participants consented to their data being included in an institutional review board-approved database study. A portion of this sample was previously analyzed in a separate study by Robinson et al (2022). More than half of the participants sought evaluation to determine eligibility for academic accommodations ( n = 181) and were referred for attention-deficit/hyperactivity disorder (ADHD; n = 137), specific learning disorder (SLD; n = 52), both ADHD and SLD ( n = 40), anxiety and/or depression ( n = 11), autism spectrum disorder ( n = 5), and other nonspecific complaints ( n = 5).…”

A large-scale investigation of the classification accuracy of various performance validity tests in a medical-legal setting.

“…The genuine memory impairment profile was not applied. Although methodological standards (Boone, 2013; Larrabee, 2014; Lippa, 2018; Sweet et al, 2021) as well as established practice (Jinkerson et al, 2023; Kulas et al, 2014; Larrabee, 2014; Ord et al, 2010; Robinson et al, 2023; Schroeder et al, 2019; Sharland et al, 2018; Tse et al, 2023) in PVT research dictate that criterion groups should be established using multiple measures, the WMT is a robust free-standing PVT containing multiple trials that sample the target construct over time. The final determination ( Pass/Fail ) is made based on three indicators.…”