Evaluation of Various Detection Strategies in the Assessment of Noncredible Memory Performance: Results of Two Experimental Studies

Abstract

Objective: This article investigates the accuracy of individual and combined indicators based on different strategies for detecting noncredible performance as part of a new test for the continuous assessment of short-term memory. Method: In two independent studies, we assessed three groups of simulators, cognitively impaired patients, and nonimpaired community members with four tasks separated by a distractor. Results: Pairwise comparisons between receiver operating characteristic (ROC) curves revealed significant differences between two clusters of indicators: mean recognition, inconsistent responses in recognition, and false positives (area under the ROC curves > .800) proved more accurate than delayed recall and false negatives (area under the ROC curves < .800) in discriminating simulators from patients. Likewise, both studies revealed that adding the false positives indicator based on cued recall to mean recognition incrementally improved classification accuracy (including sensitivity, positive predictive value, and negative predictive value ) compared with the recognition indicator alone. Conclusions: Our results support the association of two distinct indicators for the assessment of noncredible performance, of which one should be a forced-choice indicator.

A large-scale investigation of the classification accuracy of various performance validity tests in a medical-legal setting.

“…The present study examined the classification accuracy of nine PVTs (three free-standing and six embedded) administered in a large, real-world medical-legal sample. Results were broadly consistent with previous research: Free-standing PVTs had a higher classification accuracy compared to EVIs (Iverson & Binder, 2000; Lau et al, 2017; Miele et al, 2012; Sweet & Nelson, 2007), hovering around the Larrabee limit (.50 sensitivity at .90 specificity; Crişan et al, 2022; L. A. Erdodi et al, 2014).…”

Multivariate Models of Performance Validity: The Erdodi Index Captures the Dual Nature of Non-Credible Responding (Continuous and Categorical)

“…All five EVIs produced a significant AUC (range = .58–.78). The cutoffs defining the first level of failure on the EI-5 produced combinations of sensitivity (.28–.53) and specificity (.84–.93) hovering around the Larrabee limit (i.e., .50 sensitivity at .90 specificity; Crişan et al., 2021 ). Cutoffs corresponding to the second level of failure on the EI-5 disproportionately sacrificed sensitivity (.12–.32) for specificity (.95–.98).…”

That old dog can still hunt—Alternative cutoffs and the recognition trial improve the classification accuracy of the Rey 15-Item Test.

“…Using empirically based, population specific, and more liberal cutoffs greatly improves its signal detection performance, achieving a reasonable balance between sensitivity and specificity both in clinical and nonclinical settings. At optimal cutoffs (free recall ≤13 and combination score ≤25 in cognitively healthy participants; free recall ≤11 and combination score ≤23 in clinical patients), the Rey-15’s classification accuracy hovered around the “Larrabee limit”: .50 sensitivity at .90 specificity (Crişan et al, 2022; Erdodi et al, 2014). Although sensitivity at .90 specificity was generally lower for the Rey-15 (.38–.56) compared to the TOMM-1 and WCT (.47–.76) in the clinical sample, it remains a cost-effective PVT considering how inexpensive it is.…”