Examination of Performance Validity Test Failure in Relation to Number of Tests Administered

Abstract

This study examined the relationship among performance validity test (PVT) failure, number of PVTs administered, and participant characteristics including demographic, diagnostic, functional, and contextual factors in a clinical sample (N = 158) of outpatient physiatry referrals. The number of PVTs failed and the number administered showed a small non-significant correlation (rs = .13, p = .10). Participant characteristics showed associations with PVT failure consistent with prior research. A negative binomial regression model was fitted using number of PVTs failed as outcome and age, education, number of PVTs administered, clinical versus medico-legal context, and functional status as predictors. Although education and functional status were significant predictors of number of PVTs failed, the number of PVTs administered was not. A second analytic approach focused on observed false positive rates in a neurologic no-incentive (NNI) sample subset (n = 87). In contrast to a recent proposal based on statistical simulation, observed false positive rates were lower than predicted rates in NNI participants administered six, seven, or eight PVTs using a two-PVT failure cutoff. These results are interpreted as mitigating concerns that increased PVT failure is necessarily the outcome of increased PVT administration.

Cumulative False Positive Rates Given Multiple Performance Validity Tests: Commentary on Davis and Millis (2014) and Larrabee (2014)

“…Larrabee (2014) provided evidence that data from earlier publications (Larrabee 2003, 2009) show lower FPRs than predicted by either independent or correlated binomial models, and that other data (Pearson, 2009; Schroeder & Marshall, 2011; Victor, Boone, Serpa, Buehler, & Ziegler, 2009) have significantly lower FPRs than the Berthelson simulations. On the other hand, Berthelson and colleagues’ simulation data are very similar to both the empirical data of Pella and colleagues (2012), and to the empirical data of Davis and Millis (2014), as shown below. We examine details of these comparisons because they raise multiple substantive concerns about both the underlying assumptions and some of the interpretations of the data.…”

“…Davis and Millis (2014), due to concerns about the assumptions made by Berthelson and colleagues in their correlated binomial model, proposed using negative binomial regression to determine whether the number of PVTs administered was associated with the number of failed PVTs based on empirical data from a mixed sample of 158 individuals. This approach is thus a hybrid of theoretical modeling and empirical estimation.…”

“…One recent paper based on simulations suggested that widely used criteria for invalidity (e.g., two failed PVTs) may yield unacceptably high false positive rates (FPRs) if the criteria neglect to consider the number of PVTs administered and their inter-correlations (Berthelson, Mulchan, Odland, Miller, & Mittenberg, 2013). A subsequent report in The Clinical Neuropsychologist considered empirical data, which were claimed to “mitigate concerns” about multiple PVTs inflating FPRs (Davis & Millis, 2014). Following this, another report presented empirical data and interpreted these as supporting continued use of the “two failed PVTs” criterion; it was asserted further that adding PVTs does not have a significant impact on the FPR up through 7 PVTs administered (Larrabee, 2014).…”

The Structured Inventory of Malingered Symptomatology (SIMS): A Systematic Review and Meta-Analysis

“…Clinicians and researchers who decide to employ the SIMS are welladvised to bear this rule, also known as the two-failure rule (Victor et al, 2009), in mind. Concerns that the efficacy of the two-failure rule diminishes when more than two validity tests are administered (Berthelson, Mulchan, Odland, Miller, & Mittenberg, 2013) are dispelled by recent empirical findings (Davis & Millis, 2014;Larrabee, 2014).…”

Normative comparisons for large neuropsychological test batteries: User-friendly and sensitive solutions to minimize familywise false positives

“…The familywise error issue also is prominent in the normative comparisons context (e.g., Berthelson, Mulchan, Odland, Miller, & Mittenberg, 2013;Bilder, Sugar, & Hellemann, 2014;Brooks, 2010;Crawford et al, 2007;Davis & Millis, 2014;Larrabee, 2008Larrabee, , 2014Loewenstein et al, 2006;Meyers et al, 2014;Naglieri & Paolitto, 2010;Palmer, Boone, Lesser, & Wohl, 1998;Proto et al, 2014;Schretlen et al, 2008). It has been argued that in clinical practice, lack of control over familywise error in normative comparisons may result in overdiagnosis and unnecessary treatment, increasing patient burden and unnecessary costs to the health care system (Binder et al, 2009;Brooks, Iverson, Holdnack, & Feldman, 2008;Gisslén, Price, & Nilsson, 2011;Torti, Focà, Cesana, & Lescure, 2011).…”