A Bayesian approach to mixed group validation of performance validity tests.

Abstract

Mental health professionals often use structured assessment tools to help detect individuals who are feigning or exaggerating symptoms. Yet estimating the accuracy of these tools is problematic because no "gold standard" establishes whether someone is malingering or not. Several investigators have recommended using mixed group validation (MGV) to estimate the accuracy of malingering measures, but simulation studies show that typical implementations of MGV may yield vague, biased, or logically impossible results. In this article we describe a Bayesian approach to MGV that addresses and avoids these limitations. After explaining the concepts that underlie our approach, we use previously published data on the Test of Memory Malingering (TOMM; Tombaugh, 1996) to illustrate how our method works. Our findings concerning the TOMM's accuracy, which include insights about covariates such as study population and litigation status, are consistent with results that appear in previous publications. Unlike most investigations of the TOMM's accuracy, our findings neither rely on possibly flawed assumptions about subjects' intentions nor assume that experimental simulators can duplicate the behavior of real-world examinees. Our conceptual approach may prove helpful in evaluating the accuracy of many assessment tools used in clinical contexts and psycholegal determinations.

Trial 1 versus Trial 2 of the Test of Memory Malingering: Evaluating accuracy without a “gold standard”.

“…(1) Our data came from a single evaluation context. Although our findings concerning the TOMM are consistent with those of other investigators (see, e.g., Fox, 2011; Frederick & Bowden, 2009; Gervais, Rohling, Green, & Ford, 2004; Mossman et al, 2015), they do not represent definitive judgments about the performance of the TOMM. We might well have had different findings to report had we examined data from psychiatric inpatients, from individuals who had suffered demonstrably serious brain trauma, or from criminal defendants who were facing prosecution.…”

Practical challenges and methodological flexibility in prior elicitation.

“…Bolsinova et al (2017) used an elicitation procedure to specify priors for the difference in difficulty of two psychological tests. In clinical applications, Mossman et al (2015) conducted a prior elicitation for the base rate of feigning cognitive impairment in the Test of Memory Malingering, and Al-Awadhi and Garthwaite (1998) This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.…”

“…Bolsinova et al (2017) used an elicitation procedure to specify priors for the difference in difficulty of two psychological tests. In clinical applications, Mossman et al (2015) conducted a prior elicitation for the base rate of feigning cognitive impairment in the Test of Memory Malingering, and Al-Awadhi and Garthwaite (1998) elicited priors for a multivariate normal model on intelligence and memory performance after head-injuries. Recently, Sarma and Kay (2020) used prior elicitation to formulate a Bayesian model of the Balloon Analogous Risk Task (BART) that measures risk-taking behavior, and Tessler and Goodman (2019) elicited beliefs about the prevalence of certain features (e.g., being female) in semantic categories.…”

Towards Balanced VA and SSA Policies in Psychological Injury Disability Assessment

“…Mossman, Miller, Lee, Gervais, Hart, and Wygant (2015) criticized standard experimental designs to examine how well a test can identify malingerers. In simulation research, participants are asked to pretend to have impairments, but even if the results for the simulation groups involved are obtained relative to controls about a test's ability to discriminate a behavior at issue, the degree that the screening of simulated behavior can be generalized to its performance in detecting actual feigning in the forensic or clinical context is difficult to establish (Rogers & Cruise, 1998).…”