Critical item (CR) analysis expands the classification accuracy of performance validity tests based on the forced choice paradigm—Replicating previously introduced CR cutoffs within the Word Choice Test.

Abstract

Objective: This study was designed to replicate previous research on critical item analysis within the Word Choice Test (WCT). Method: Archival data were collected from a mixed clinical sample of 119 consecutively referred adults (Mage = 51.7, Meducation = 14.7). The classification accuracy of the WCT was calculated against psychometrically defined criterion groups. Results: Critical item analysis identified an additional 2%–5% of the sample that passed traditional cutoffs as noncredible. Passing critical items after failing traditional cutoffs was associated with weaker independent evidence of invalid performance, alerting the assessor to the elevated risk for false positives. Failing critical items in addition to failing select traditional cutoffs increased overall specificity. Non-White patients were 2.5 to 3.5 times more likely to Fail traditional WCT cutoffs, but select critical item cutoffs limited the risk to 1.5–2. Conclusions: Results confirmed the clinical utility of critical item analysis. Although the improvement in sensitivity was modest, critical items were effective at containing false positive errors in general, and especially in racially diverse patients. Critical item analysis appears to be a cost-effective and equitable method to improve an instrument’s classification accuracy.

Combining accuracy scores with time cutoffs improves the specificity of the Word Choice Test.

“…Note. AUC = area under the curve; CI = confidence interval; WCT = Word Choice Test; VI-7 = Validity Index Seven (pass defined as ≤1; fail defined as ≥2); CR = critical item analysis (Erdodi, Tyson, et al, 2018;Holcomb et al, 2022); T2C = time to completion (s); COMB = combined cutoffs (pass defined as passing at least one of the cutoff; fail defined as failing both cutoffs); BR Fail = base rate of failure (% of the sample that failed a given cutoff) Finally, a T2C cutoff of ≥171" produced a good combination of sensitivity (.40) and specificity (.86) at .797 OCC. Increasing the cutoff to ≥190" maintained sensitivity while improving both specificity (.93) and OCC (.855).…”

Utility of the D-KEFS color–word interference test as an embedded validity indicator in psychoeducational evaluations.

“…Although similar in size to previous investigations, replication in a larger study with participants from more than one geographic background and greater racial heterogeneity is warranted to increase the reliability of our findings. This is particularly important as recent studies have shown that racially diverse individuals may be at greater risk for false positives (Holcomb et al, 2022). We were also unable to compare the ADHD and SLD subsamples due to not having enough individuals in each of these groups.…”

Examining the effect of repeat administration, alternate versions, and performance validity on letter fluency tests in a mixed clinical sample.

“…This article is intended solely for the personal use of the individual user and is not to be disseminated broadly. Heaton et al (2004); FS-PVT-4 =number of failures on the following free-standing performance validity tests: Rey-15 Item Test free recall (Fail defined as ≤11; Ashendorf et al, 2021;Poynter et al, 2019); first trial of the Test of Memory Malingering (Fail defined as ≤43; Denning, 2012;Erdodi, 2022;Jones, 2013;Kulas et al, 2014;Rai & Erdodi, 2021); Warrington's Recognition Memory Test-Words (Fail defined as accuracy ≤42; or time-tocompletion ≥207 s; Erdodi et al, 2014;Kim et al, 2010;Zuccato et al, 2018); Word Choice Test (Fail defined as accuracy ≤45; or time-to-completion ≥171 s; Cutler, Greenacre, et al, 2022;Erdodi, 2021;Holcomb et al, 2022;Zuccato et al, 2018); FS-PVT-4 ≤ 1 was defined as valid; FS-PVT-4 ≥ 2 was defined as invalid; EI-5 VER = Erdodi Index Five-verbal (Fail defined as ≥4); EI-5 PSP = Erdodi Index Five-processing speed (Fail defined as ≥4).…”

“…This article is intended solely for the personal use of the individual user and is not to be disseminated broadly. Denning, 2012;Erdodi, 2022;Jones, 2013;Kulas et al, 2014;Rai & Erdodi, 2021); Warrington's Recognition Memory Test-Words (Fail defined as accuracy ≤42; or time-to-completion ≥207 s; Erdodi et al, 2014;Kim et al, 2010;Zuccato et al, 2018); Word Choice Test (Fail defined as accuracy ≤45; or time-to-completion ≥171 s; Cutler, Greenacre, et al, 2022;Erdodi, 2021;Holcomb et al, 2022;Zuccato et al, 2018); FS-PVT-4 ≤ 1 was defined as valid; FS-PVT-4 ≥ 2 was defined as invalid; EI-5 VER = Erdodi Index Five-verbal (Fail defined as ≥4); EI-5 PSP = Erdodi Index Fiveprocessing speed (Fail defined as ≥4); T = demographically adjusted T scores (M = 50, SD = 10) based on norms by Heaton et al (2004); phCLS = number of phonemic clusters; d AVG = average effect size across the four scores within a given version of the letter fluency task (FAS vs. CFL); PVT = performance validity tests.…”

“…However, Merten and Merckelbach (2013) cautioned against using psychogenic interference as a superordinate category to noncredible presentation (i.e., absolve noncredible responding due to the presence of high levels of symptom report). In the context of PVTs, the construct is likely contaminated by noncredible responding (i.e., symptom overreport; Boskovic et al, 2020; Holcomb et al, 2022; Morris et al, 2022; Ord, Miskey, et al, 2021), response to treatment (Jurick et al, 2020), and certain personality traits (Menon et al, 2018).…”