Considerations underlying the use of mixed group validation.

Abstract

Mixed Group Validation (MGV) is an approach for estimating the diagnostic accuracy of tests. MGV is a promising alternative to the more commonly used Known Groups Validation (KGV) approach for estimating diagnostic accuracy. The advantage of MGV lies in the fact that the approach does not require a perfect external validity criterion or gold standard. However, the research designs where MGV is most appropriate have not been thoroughly explored. We give a brief description of the ideal research design to minimize error for MGV studies, test whether the MGV assumptions hold with clinical data, evaluate whether there is evidence of assumption violation among published MGV studies, give a practical description of the MGV assumptions, and describe an example of an optimal use of MGV. Ultimately, we conclude that MGV is not generally superior to KGV but may be used in some cases where the assumptions and standard error have been considered appropriately.

Diagnostic Test Score Validation With a Fallible Criterion

“…The problem that reference tests are rarely infallible has long been recognized, and the first method popularized in psychology as an alternative was introduced by Dawes and Meehl (1966). More recently, Mixed Group Validation and related methods have received a resurgence of interest in psychology (e.g., Frederick & Bowden, 2009a, 2009b; Jewsbury & Bowden, 2013, 2014; Mossman et al, 2015; Mossman, Wygant, & Gervais, 2012; Ortega, Labrenz, Markowitsch, & Piefke, 2013; Thomas, Lanyon, & Millsap, 2009; Tolin, Steenkamp, Marx, & Litz, 2010).…”

“…Mixed Group Validation studies are similar to Known Group Validation studies, in that they have a test to be validated, a reference test, and a construct. However, instead of assuming the reference test is infallible, the defining assumption of Mixed Group Validation is that the reference test and test to be validated are conditionally independent given the true construct status (Dawes & Meehl, 1966; Jewsbury & Bowden, 2013). That is,…”

“…Mixed Group Validation assumptions of conditional independence may be inappropriate for many or most psychological diagnostic test validation studies (Jewsbury & Bowden, 2013). For example, the reference test and the test to be validated will often both be sensitive to symptom severity, over and above sensitivity to the dichotomized construct, which is a type of conditional dependence.…”

“…Indeed, the test to be validated and the reference test may often be expected to share sensitivity to several other non-construct effects, such as effort, method effects, reading comprehension, cognitive abilities, response styles, and test taker sophistication. Mixed Group Validation is not robust to assumption violation (Jewsbury & Bowden, 2014), and many of the existing Mixed Group Validation studies are demonstrably and severely biased due to assumption violation (Jewsbury & Bowden, 2013).…”

“…The Mixed Group Validation assumptions are especially implausible if the construct is an arbitrary dichotomization of a continuous dimension or has underlying continuous attributes (Jewsbury & Bowden, 2013), which may be true for many psychopathological constructs (Widiger & Samuel, 2005). However, models with an alternate conceptualization of the construct and corresponding assumptions may still be appropriate for such constructs, and several such methods are explored in the present article.…”