Multivariate models provide an effective psychometric solution to the variability in classification accuracy of D-KEFS Stroop performance validity cutoffs

Cutler, Laura; Greenacre, Matthew; Abeare, Christopher A.; Sirianni, Christina D.; Roth, Robert M.; Erdődi, László A.

doi:10.1080/13854046.2022.2073914

Cited by 25 publications

(29 citation statements)

References 142 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The failure to replicate the high specificity associated with an accuracy score of ≤47 (An et al, 2019; Barhon et al, 2015; Cutler et al, 2023) may be due to the lack of memory-based free-standing PVTs to serve as the criterion and thus, capitalize on the congruence between the predictor and criterion measures (Erdodi et al, 2019; Lace et al, 2020; Rai & Erdodi, 2021; Schroeder et al, 2019). In addition, the first two studies were based on the experimental malingering paradigm, which is prone to producing inflated estimates of a PVTs signal detection performance (C.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Combining accuracy scores with time cutoffs improves the specificity of the Word Choice Test.

Tyson¹,

Shahein²

2023

Psychology & Neuroscience

View full text Add to dashboard Cite

Objective: This study was designed to explore the classification accuracy of various Word Choice Test (WCT) cutoffs. Method: Archival data were collected from a consecutive case series of a mixed sample of 79 patients clinically referred for neuropsychological assessment. Criterion groups were established using a combination of seven embedded validity indicators. Results: A WCT accuracy score of ≤45 emerged as the optimal accuracy cutoff (.47 sensitivity at .92 specificity). A time cutoff of ≥190" achieved a good combination of sensitivity (.40) and specificity (.93). Critical item analysis produced high specificity (.93-.98) but low and variable sensitivity (.19-.44). Combining the accuracy and time cutoffs resulted in perfect specificity, correctly classifying 91% of the sample. Conclusion: Imposing highly conservative accuracy cutoffs on the WCT unnecessarily sacrifices sensitivity. Recruiting alternative detection methods (completion time and critical items) seems a better approach, as it minimizes false positives while maximizing overall classification accuracy. Public Significance StatementThis study was designed to explore the classification accuracy of various cutoff scores on the WCT, a performance validity test used to differentiate credible and noncredible cognitive performance in patients undergoing neuropsychological assessment. Using a mixed clinical sample of 79 patients, our study identified optimal WCT cutoff scores for accuracy, time-to-complete, and critical item analysis. These findings demonstrate that multiple performance validity detection methods within the WCT can help maximize the diagnostic accuracy of neuropsychological assessment.

show abstract

Section: Discussionmentioning

confidence: 99%

“…A T2C cutoff of ≥171” was associated with .49 sensitivity at .91 specificity and correctly identified an additional 6%–10% of noncredible response sets missed by accuracy cutoffs. A subsequent replication by Cutler et al (2023) reported slightly better classification accuracy with a T2C ≥ 171” associated with .67 sensitivity at .96 specificity.…”

mentioning

confidence: 91%

Combining accuracy scores with time cutoffs improves the specificity of the Word Choice Test.

Tyson¹,

Shahein²

2023

Psychology & Neuroscience

View full text Add to dashboard Cite

show abstract

“…This article is intended solely for the personal use of the individual user and is not to be disseminated broadly. Ashendorf et al, 2017;; WORD D-KEFS = word reading subtest of the Delis-Kaplan executive function system (age-corrected scaled score; Cutler, Greenacre, et al, 2022;.…”

Section: Administration Timementioning

confidence: 99%

“…This article is intended solely for the personal use of the individual user and is not to be disseminated broadly. Denning, 2012;Erdodi, 2022;Jones, 2013;Kulas et al, 2014;Rai & Erdodi, 2021); Warrington's Recognition Memory Test-Words (Fail defined as accuracy ≤42; or time-to-completion ≥207 s; Erdodi et al, 2014;Kim et al, 2010;Zuccato et al, 2018); Word Choice Test (Fail defined as accuracy ≤45; or time-to-completion ≥171 s; Cutler, Greenacre, et al, 2022;Erdodi, 2021;Holcomb et al, 2022;Zuccato et al, 2018); FS-PVT-4 ≤ 1 was defined as valid; FS-PVT-4 ≥ 2 was defined as invalid; EI-5 VER = Erdodi Index Five-verbal (Fail defined as ≥4); EI-5 PSP = Erdodi Index Fiveprocessing speed (Fail defined as ≥4); T = demographically adjusted T scores (M = 50, SD = 10) based on norms by Heaton et al (2004); phCLS = number of phonemic clusters; d AVG = average effect size across the four scores within a given version of the letter fluency task (FAS vs. CFL); PVT = performance validity tests.…”

Section: Main Findingsmentioning

confidence: 99%

Examining the effect of repeat administration, alternate versions, and performance validity on letter fluency tests in a mixed clinical sample.

Boucher¹,

May²,

Shahein³

et al. 2023

Psychology & Neuroscience

Self Cite

View full text Add to dashboard Cite

Objective: This study was designed to compare different versions of the letter fluency test (FAS/CFL), evaluate the effect of repeat administration and the classification accuracy of traditional and novel cutoffs as performance validity tests (PVTs). Method: Archival data were collected from a mixed clinical sample (n = 64) of patients physician-referred for neuropsychological assessment (M Age = 43.2; M Education = 14.1). Sensitivity and specificity of various letter fluency cutoffs were calculated against various psychometric definitions of invalid performance. Results: There was no difference between the FAS and CFL on total score and phonemic clusters. Learning effects were small and limited to the CFL. A T score of ≤33 on the FAS and ≤35 on CFL was specific (.89-.98), but relatively insensitive (.27-.45). Likewise, ≤4 phonemic clusters were associated with high specificity (.88-.94) but variable sensitivity (.22-.60) and were orthogonal to self-reported anxiety and depression. A raw score of ≤7 (FAS) or ≤8 (CFL) on the trial F had comparable classification accuracy (.18-.44 sensitivity at .91-.98 specificity). Failure on multiple validity cutoffs had near-ceiling specificity (.92-1.00). Declining output was similar in valid and invalid response sets. Conclusions: FAS and CFL are psychometrically equivalent versions of the letter fluency test and are relatively robust to learning effects on serial testing. Both can function as effective embedded PVTs, although they are limited by low and variable sensitivity. Process variables had mixed results as predictors of performance validity. Public Significance StatementDifferent versions of the same test are expected to produce similar outcomes and repeat administrations often result in higher performance. However, these assumptions are rarely verified empirically. This study found that two versions of the letter fluency test (FAS and CFL) produce converging results, learning effects in clinical patients are small, and both versions function well as performance validity indicators.

show abstract

“…The application of the proposed method shows a high prediction capacity. Also, Cutler et al (11) used multivariate models as an effective psychometric solution to the variability in classification accuracy of D-KEFS Stroop performance validity cutoffs (performance validity tests during neuropsychological assessments). The results indicate that the multivariate approach to performance validity assessment provides a methodological safeguard against sample-and instrumentspecific fluctuations in classification accuracy, striking a reasonable balance between sensitivity and specificity.…”

Section: Introductionmentioning

confidence: 99%

Multivariate analysis in data science for the geospatial distribution of the breast cancer mortality rate in Colombia

et al. 2023

View full text Add to dashboard Cite

This research is framed in the area of biomathematics and contributes to the epidemiological surveillance entities in Colombia to clarify how breast cancer mortality rate (BCM) is spatially distributed in relation to the forest area index (FA) and circulating vehicle index (CV). In this regard, the World Health Organization has highlighted the scarce generation of knowledge that relates mortality from tumor diseases to environmental factors. Quantitative methods based on geospatial data science are used with cross-sectional information from the 2018 census; it’s found that the BCM in Colombia is not spatially randomly distributed, but follows cluster aggregation patterns. Under multivariate modeling methods, the research provides sufficient statistical evidence in terms of not rejecting the hypothesis that if a spatial unit has high FA and low CV, then it has significant advantages in terms of lower BCM.

show abstract

Multivariate models provide an effective psychometric solution to the variability in classification accuracy of D-KEFS Stroop performance validity cutoffs

Cited by 25 publications

References 142 publications

Combining accuracy scores with time cutoffs improves the specificity of the Word Choice Test.

Combining accuracy scores with time cutoffs improves the specificity of the Word Choice Test.

Examining the effect of repeat administration, alternate versions, and performance validity on letter fluency tests in a mixed clinical sample.

Multivariate analysis in data science for the geospatial distribution of the breast cancer mortality rate in Colombia

Contact Info

Product

Resources

About