Geographic Variation and Instrumentation Artifacts: in Search of Confounds in Performance Validity Assessment in Adults with Mild TBI

Lichtenstein, Jonathan D.; Greenacre, Matthew; Cutler, Laura; Abeare, Kaitlyn; Baker, Shannon D.; Kent, Katrina J.; Ali, Sami; Erdődi, László A.

doi:10.1007/s12207-019-09354-w

Cited by 33 publications

(18 citation statements)

References 135 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This article is intended solely for the personal use of the individual user and is not to be disseminated broadly. (Hurtubise et al, 2020); Coding Age-Corrected Scaled Score (ACSS) ≤5 (Ashendorf et al, 2017;Erdodi & Abeare, 2020); complex ideational material T ≤ 29 ; digit span ACSS ≤ 6 (Erdodi & Abeare, 2020;Hurtubise et al, 2020); letter fluency (FAS) T ≤ 31 ; forced choice recognition trial of the California verbal learning ≤15 ; logical memory recognition raw score ≤20 (Dunn et al, 2021;Erdodi, 2021;Pearson, 2009); symbol search ACSS ≤ 5 (Erdodi & Abeare, 2020); Trails A T ≤ 33 (Abeare et al, 2019;Ashendorf et al, 2017;Erdodi, 2021); EI-5 = Erdodi Index Five; CR = critical items; LR = likelihood ratio; B/P = Borderline over Pass; F/P = Fail over Pass; F/B = Fail over Borderline. The actual combination of critical items is not disclosed here to protect test security.…”

Section: Demographic Factors and Pvt Failurementioning

confidence: 99%

Critical item (CR) analysis expands the classification accuracy of performance validity tests based on the forced choice paradigm—Replicating previously introduced CR cutoffs within the Word Choice Test.

Holcomb¹,

Roth²,

Tyson³

et al. 2022

Neuropsychology

Self Cite

View full text Add to dashboard Cite

Objective: This study was designed to replicate previous research on critical item analysis within the Word Choice Test (WCT). Method: Archival data were collected from a mixed clinical sample of 119 consecutively referred adults (Mage = 51.7, Meducation = 14.7). The classification accuracy of the WCT was calculated against psychometrically defined criterion groups. Results: Critical item analysis identified an additional 2%–5% of the sample that passed traditional cutoffs as noncredible. Passing critical items after failing traditional cutoffs was associated with weaker independent evidence of invalid performance, alerting the assessor to the elevated risk for false positives. Failing critical items in addition to failing select traditional cutoffs increased overall specificity. Non-White patients were 2.5 to 3.5 times more likely to Fail traditional WCT cutoffs, but select critical item cutoffs limited the risk to 1.5–2. Conclusions: Results confirmed the clinical utility of critical item analysis. Although the improvement in sensitivity was modest, critical items were effective at containing false positive errors in general, and especially in racially diverse patients. Critical item analysis appears to be a cost-effective and equitable method to improve an instrument’s classification accuracy.

show abstract

Section: Demographic Factors and Pvt Failurementioning

confidence: 99%

Critical item (CR) analysis expands the classification accuracy of performance validity tests based on the forced choice paradigm—Replicating previously introduced CR cutoffs within the Word Choice Test.

Holcomb¹,

Roth²,

Tyson³

et al. 2022

Neuropsychology

Self Cite

View full text Add to dashboard Cite

show abstract

“…Clinical classifications (MCI and AD) were not confirmed by a treating physician. The sample is geographically limited to a single region of the United States, and predominantly Caucasian, which has been shown to influence assessment outcomes (Lichtenstein et al, 2019). Also, some of the subsamples were small.…”

Section: Discussionmentioning

confidence: 99%

Increasing the cutoff on the MMSE and DRS-2 improves clinical classification accuracy in highly educated older adults.

Erdődi¹,

Shahein²,

Fareez³

et al. 2020

Psychology & Neuroscience

View full text Add to dashboard Cite

Cognitive reserve could mask the deleterious effects of neurodegenerative disorders in older adults with high premorbid functioning, resulting in false negatives on cognitive screening tests. Failing to detect early signs of cognitive decline may result in missing a critical period of intervention with disease modifying drugs or making informed decisions about end of life issues. The Mini-Mental State Exam (MMSE) and the Dementia Rating Scale, 2nd edition (DRS-2) were compared in a sample of 113 highly educated older adults from northern New England recruited for a research study. Participants were classified as cognitively intact or impaired by a panel of experts based on neuroimaging results, psychometric testing, clinical history, and self-reported level of adaptive functioning corroborated by collateral informants. The 2 instruments produced comparably high specificity, but variable sensitivity to cognitive impairment. Surprisingly, the MMSE consistently outperformed the DRS-2 in overall classification accuracy. Raising the standard cutoffs improved the signal detection performance of both tests with minimal loss in specificity and thus, appears to be a clinically justifiable trade-off. At around 90% specificity, MMSE Ͻ28 and DRS-2 Ͻ139 correctly identified 86% and 67% of the sample. When these cutoffs were restricted to the detection of mild cognitive deficit only, sensitivity declined slightly (81% and 57%, respectively). Neuropsychological tests of memory and executive function were more sensitive to cognitive decline than measures of attention and processing speed. Findings suggest that higher cutoffs may be warranted, and perhaps necessary, in examinees with high educational achievement. At the proposed alternative cutoffs, the MMSE and DRS-2 remained sensitive to even subclinical cognitive deficits. Replication with physician-referred patients is needed to establish the generalizability of the findings to clinical settings.

show abstract

“…Specifically, its generalizability to patients with unique presentations (e.g., psychiatric inpatients; patients undergoing chemotherapy; patients with dementia, chronic pain, sports concussion, or limited English proficiency) must be established before the V-5 can be recommended for widespread use. Neuropsychiatric constructs have been reported to vary across assessment settings (Abeare, Messa, et al, 2019; An, Kaploun, Erdodi, & Abeare, 2017, 2019; Bailey, Soble, & O’Rourke, 2018; Erdodi & Lichtenstein, 2017; Glassmire et al, 2019; Poynter et al, 2019; Roye, Calamia, Bernstein, De Vito, & Hill, 2019) and locations (Eglit et al, 2019; Erdodi, Abeare, et al, 2018; Lichtenstein et al, 2019). Therefore, readers are reminded that the V-5 is currently at an early stage of validation and is awaiting independent replication.…”

Section: Discussionmentioning

confidence: 99%

“…In addition, all patients were Caucasian native speakers of English, raising questions about generalizability to individuals with different demographic characteristics. Given recent concerns about geographic variation in psychological constructs (Lichtenstein et al, 2019;Roth, Erdodi, McCulloch, & Isquith, 2015), no definitive conclusion can be reached at this time about the V-5's clinical utility. Participants in both samples were cognitively high functioning and had aboveaverage reading skills.…”

Section: Strengths Limitations and Direction For Future Researchmentioning

confidence: 98%

Introducing a Five-Variable Psychiatric Screener based on the Visual Analog Scale (V-5).

Erdődi¹,

Sabelli²,

An³

et al. 2020

Psychology & Neuroscience

Self Cite

View full text Add to dashboard Cite

This study was designed to provide initial empirical validation for the Five-Variable Psychiatric Screener (V-5), a five-variable (energy, depression, anxiety, pain, and fatigue) psychiatric screener based on the visual analog scaling method. The V-5 was administered twice (at the beginning and end of a neuropsychological battery) to 102 clinically referred patients and 32 student volunteers, in conjunction with established self-reported symptom inventories designed to measure depression, anxiety, and somatic concerns to serve as criterion variables. The V-5 was sensitive to time-related changes in mental stamina. It also effectively differentiated between the clinical and student samples. Finally, it was highly predictive of the criterion measures (area under the curve [AUC] = .62–.99). The proposed V-5 cutoffs for mild, moderate, and severe symptoms (≥20, ≥40, and ≥70) produced favorable classification accuracy. The generalizability of the findings is limited by the sample size and the choice of criterion measures. Replication in larger samples with different criterion tests is needed to establish the ecological validity of the V-5. Data from multiple analyses support the construct validity and clinical utility of the V-5. Given its low cost, quick and easy administration/scoring (<1 min), and minimal demands on the examinee (reading level, cognitive ability, motivation), the V-5 has the potential to become a cost-effective and empirically validated rapid assessment instrument in clinical and research settings requiring large-scale repeated testing. The generalizability of the V-5 must be evaluated through replication in geographically, demographically, clinically, and linguistically diverse samples before its widespread use can be recommended.

show abstract

Geographic Variation and Instrumentation Artifacts: in Search of Confounds in Performance Validity Assessment in Adults with Mild TBI

Cited by 33 publications

References 135 publications

Critical item (CR) analysis expands the classification accuracy of performance validity tests based on the forced choice paradigm—Replicating previously introduced CR cutoffs within the Word Choice Test.

Critical item (CR) analysis expands the classification accuracy of performance validity tests based on the forced choice paradigm—Replicating previously introduced CR cutoffs within the Word Choice Test.

Increasing the cutoff on the MMSE and DRS-2 improves clinical classification accuracy in highly educated older adults.

Introducing a Five-Variable Psychiatric Screener based on the Visual Analog Scale (V-5).

Contact Info

Product

Resources

About